capreolus.benchmark

Package Contents

Classes

Benchmark(config=None, provide=None, share_dependency_objects=False, build=True) the module base class
DummyBenchmark(config=None, provide=None, share_dependency_objects=False, build=True) the module base class
WSDM20Demo(config=None, provide=None, share_dependency_objects=False, build=True) Robust04 benchmark equivalent to robust04.yang19
Robust04Yang19(config=None, provide=None, share_dependency_objects=False, build=True) Robust04 benchmark using the folds from Yang et al. [1]
ANTIQUE(config=None, provide=None, share_dependency_objects=False, build=True) A Non-factoid Question Answering Benchmark from Hashemi et al. [1]
MSMarcoPassage(config=None, provide=None, share_dependency_objects=False, build=True) the module base class
CodeSearchNetCorpus(config=None, provide=None, share_dependency_objects=False, build=True) the module base class
CodeSearchNetChallenge(config=None, provide=None, share_dependency_objects=False, build=True) CodeSearchNetChallenge can only be used for training but not for evaluation since qrels is not provided
COVID(config=None, provide=None, share_dependency_objects=False, build=True) Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit
CovidQA(config=None, provide=None, share_dependency_objects=False, build=True) the module base class
capreolus.benchmark.logger[source]
capreolus.benchmark.PACKAGE_PATH[source]
class capreolus.benchmark.Benchmark(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: profane.ModuleBase

the module base class

module_type = benchmark[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type[source]
relevance_level = 1[source]
qrels[source]
topics[source]
folds[source]
class capreolus.benchmark.DummyBenchmark(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

the module base class

module_name = dummy[source]
dependencies[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type = title[source]
class capreolus.benchmark.WSDM20Demo(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

Robust04 benchmark equivalent to robust04.yang19

module_name = wsdm20demo[source]
dependencies[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type = title[source]
class capreolus.benchmark.Robust04Yang19(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

Robust04 benchmark using the folds from Yang et al. [1]

[1] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. SIGIR 2019.

module_name = robust04.yang19[source]
dependencies[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type = title[source]
class capreolus.benchmark.ANTIQUE(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

A Non-factoid Question Answering Benchmark from Hashemi et al. [1]

[1] Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W. Bruce Croft. 2020. ANTIQUE: A non-factoid question answering benchmark. ECIR 2020.

module_name = antique[source]
dependencies[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type = title[source]
relevance_level = 2[source]
class capreolus.benchmark.MSMarcoPassage(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

the module base class

module_name = msmarcopassage[source]
dependencies[source]
qrel_file[source]
topic_file[source]
fold_file[source]
query_type = title[source]
class capreolus.benchmark.CodeSearchNetCorpus(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

the module base class

module_name = codesearchnet_corpus[source]
dependencies[source]
url = https://s3.amazonaws.com/code-search-net/CodeSearchNet/v2[source]
query_type = title[source]
file_fn[source]
qrel_dir[source]
topic_dir[source]
fold_dir[source]
qidmap_dir[source]
docidmap_dir[source]
config_spec[source]
qid_map[source]
docid_map[source]
build(self)[source]
download_if_missing(self)[source]
get_docid(self, url, code_tokens)[source]

retrieve the doc id according to the doc dict

class capreolus.benchmark.CodeSearchNetChallenge(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

CodeSearchNetChallenge can only be used for training but not for evaluation since qrels is not provided

module_name = codesearchnet_challenge[source]
dependencies[source]
config_spec[source]
url = https://raw.githubusercontent.com/github/CodeSearchNet/master/resources/queries.csv[source]
query_type = title[source]
file_fn[source]
topic_file[source]
qid_map_file[source]
download_if_missing(self)[source]

download query.csv and prepare queryid - query mapping file

class capreolus.benchmark.COVID(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit

module_name = covid[source]
dependencies[source]
data_dir[source]
topic_url = https://ir.nist.gov/covidSubmit/data/topics-rnd%d.xml[source]
qrel_url = https://ir.nist.gov/covidSubmit/data/qrels-rnd%d.txt[source]
lastest_round = 3[source]
config_spec[source]
build(self)[source]
download_if_missing(self)[source]
xml2trectopic(self, xmlfile)[source]
class capreolus.benchmark.CovidQA(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

the module base class

module_name = covidqa[source]
dependencies[source]
url = https://raw.githubusercontent.com/castorini/pygaggle/master/data/kaggle-lit-review-%s.json[source]
available_versions = ['0.1', '0.2'][source]
datadir[source]
config_spec[source]
build(self)[source]
download_if_missing(self)[source]