Running Pipelines with the CLI¶
Capreolus provides a command line interface for running experiments using pipelines that are described by
Task modules. To create a new pipeline, you’ll need to create a new
Task before using the CLI.
Capreolus takes a functional approach to describing an experiment. An experiment is simply a pipeline plus a set of configuration options specifying both classes to use for the pipeline’s modules and configuration options associated with each module. These configuration options fully and deterministically describe the pipeline; the output should always be the same given the same configuration options (modulo any CUDA non-determinism). Capreolus takes advantage of this functional approach to cache intermediate outputs (given module dependencies).
The CLI takes a pipeline to run, such as
rank.searcheval, and optionally a list of configuration options for the pipeline:
capreolus <pipeline> [with <configuration options>].
The first part of the pipeline corresponds to a Task (
rank) and the second part corresponds to one of the Task’s commands (
searcheval, which runs
search followed by
If no command is specified, a default chosen by the Task is run.
Configuration options are specified in
All Tasks provide several commands to help understand their operation.
print_config command displays the Task’s configuration, including any options specified on the command line.
print_pipeline command displays the pipeline’s dependency graph, including current module choices.
modules Task provides list of all module types and classes that are currently registered. For example:
$ capreolus modules module type=benchmark name=antique name=dummy name=robust04.yang19 name=nf ... module type=reranker name=CDSSM name=ConvKNRM name=DRMM name=DSSM ...
Results and cached objects are stored in
~/.capreolus/cache/ by default. Set the
CAPREOLUS_CACHE environment variables to change these locations. For example:
RankTaskwith the NFCorpus
Benchmark. This will download the collection, create an index, and search for the NFCorpus topics. The
Benchmarkspecifies a dependency on
collection.name=nfand provides the corresponding topics and relevance judgments.
$ capreolus rank.searcheval with searcher.name=BM25 \ searcher.index.stemmer=porter benchmark.name=nf
Use a similar pipeline, but with RM3 query expansion and a small grid search over expansion parameters. The evaluation command will report cross-validated results using the folds specified by the
$ capreolus rank.searcheval with \ searcher.index.stemmer=porter benchmark.name=nf \ searcher.name=BM25RM3 searcher.fbDocs=5-10-15 searcher.fbTerms=5-25-50
RerankTaskto run the same
RankTaskpipeline optimized for recall@1000, and then train a
Rerankeroptimized for P@20 on the first fold provided by the
Benchmark. We limit training to two iterations (
niters) of size
itersizeto keep the training process from taking too long.
$ capreolus rerank.traineval with \ rank.searcher.index.stemmer=porter benchmark.name=nf \ rank.searcher.name=BM25RM3 rank.searcher.fbDocs=5-10-15 rank.searcher.fbTerms=5-25-50 \ rank.optimize=recall_1000 reranker.name=KNRM reranker.trainer.niters=2 optimize=P_20
ReRerankTaskdemonstrates pipeline flexibility by adding a second reranking step on top of the output from
capreolus rererank.print_configto see the configuration options it expects. (Hint: it consists of a
rankas before, followed by a
rerank1, followed by another