capreolus.benchmark.robust04

Module Contents

Classes

Robust04	Robust04 benchmark using the title folds from Huston and Croft. [1] Each of these is used as the test set.
Robust04Yang19	Robust04 benchmark using the folds from Yang et al. [1]
Robust04Yang19Desc	Robust04 benchmark using the folds from Yang et al. [1]
Robust04Huston14	Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.
Robust04Huston14Desc	Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Attributes

PACKAGE_PATH

capreolus.benchmark.robust04.PACKAGE_PATH

class capreolus.benchmark.robust04.Robust04(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: capreolus.benchmark.Benchmark

Robust04 benchmark using the title folds from Huston and Croft. [1] Each of these is used as the test set. Given the remaining four folds, we split them into the same train and dev sets used in recent work. [2]

[1] Samuel Huston and W. Bruce Croft. 2014. Parameters learned in the comparison of retrieval models using term dependencies. Technical Report.

[2] Sean MacAvaney, Andrew Yates, Arman Cohan, Nazli Goharian. 2019. CEDR: Contextualized Embeddings for Document Ranking. SIGIR 2019.

module_name = 'robust04'

dependencies

qrel_file

topic_file

fold_file

query_type = 'title'

class capreolus.benchmark.robust04.Robust04Yang19(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: capreolus.benchmark.Benchmark

Robust04 benchmark using the folds from Yang et al. [1]

[1] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. SIGIR 2019.

module_name = 'robust04.yang19'

dependencies

qrel_file

topic_file

fold_file

query_type = 'title'

class capreolus.benchmark.robust04.Robust04Yang19Desc(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: Robust04Yang19, capreolus.benchmark.Benchmark

Robust04 benchmark using the folds from Yang et al. [1]

[1] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. SIGIR 2019.

module_name = 'robust04.yang19.desc'

query_type = 'desc'

class capreolus.benchmark.robust04.Robust04Huston14(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: capreolus.benchmark.Benchmark

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:

• a topics dict mapping query ids (qids) to queries

• a qrels dict mapping qids to docids and relevance labels

• a folds dict mapping a fold name to training, dev (validation), and testing qids

• if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = 'robust04.huston14.title'

dependencies

qrel_file

topic_file

fold_file

query_type = 'title'

class capreolus.benchmark.robust04.Robust04Huston14Desc(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: Robust04Huston14, capreolus.benchmark.Benchmark

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:

• a topics dict mapping query ids (qids) to queries

• a qrels dict mapping qids to docids and relevance labels

• a folds dict mapping a fold name to training, dev (validation), and testing qids

• if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly

module_name = 'robust04.huston14.desc'

fold_file

query_type = 'desc'