capreolus.benchmark.cds

Module Contents

Classes

CDS

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

CDSDesc

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Attributes

PACKAGE_PATH

capreolus.benchmark.cds.PACKAGE_PATH[source]
class capreolus.benchmark.cds.CDS(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.IRDBenchmark

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:
  • a topics dict mapping query ids (qids) to queries

  • a qrels dict mapping qids to docids and relevance labels

  • a folds dict mapping a fold name to training, dev (validation), and testing qids

  • if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = cds[source]
ird_dataset_names = ['pmc/v1/trec-cds-2014', 'pmc/v1/trec-cds-2015', 'pmc/v2/trec-cds-2016'][source]
dependencies[source]
fold_file[source]
query_type = summary[source]
query_types[source]
build(self)[source]
ird_load_qrels(self)[source]
ird_load_topics(self)[source]
class capreolus.benchmark.cds.CDSDesc(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: CDS

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:
  • a topics dict mapping query ids (qids) to queries

  • a qrels dict mapping qids to docids and relevance labels

  • a folds dict mapping a fold name to training, dev (validation), and testing qids

  • if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = cds.desc[source]
query_type = description[source]