capreolus.benchmark.core18

Module Contents

Classes

Core18	Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.
Core18Desc	Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Attributes

PACKAGE_PATH

capreolus.benchmark.core18.PACKAGE_PATH

class capreolus.benchmark.core18.Core18(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: capreolus.benchmark.IRDBenchmark

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:

• a topics dict mapping query ids (qids) to queries

• a qrels dict mapping qids to docids and relevance labels

• a folds dict mapping a fold name to training, dev (validation), and testing qids

• if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = 'core18'

query_type = 'title'

ird_dataset_names = ['wapo/v2/trec-core-2018']

dependencies

fold_file

class capreolus.benchmark.core18.Core18Desc(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: Core18

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:

• a topics dict mapping query ids (qids) to queries

• a qrels dict mapping qids to docids and relevance labels

• a folds dict mapping a fold name to training, dev (validation), and testing qids

• if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = 'core18.desc'

query_type = 'desc'

fold_file