capreolus.benchmark
¶
Submodules¶
Package Contents¶
Classes¶
Benchmark (config=None, provide=None, share_dependency_objects=False, build=True) |
the module base class |
DummyBenchmark (config=None, provide=None, share_dependency_objects=False, build=True) |
the module base class |
WSDM20Demo (config=None, provide=None, share_dependency_objects=False, build=True) |
Robust04 benchmark equivalent to robust04.yang19 |
Robust04Yang19 (config=None, provide=None, share_dependency_objects=False, build=True) |
Robust04 benchmark using the folds from Yang et al. [1] |
ANTIQUE (config=None, provide=None, share_dependency_objects=False, build=True) |
A Non-factoid Question Answering Benchmark from Hashemi et al. [1] |
MSMarcoPassage (config=None, provide=None, share_dependency_objects=False, build=True) |
the module base class |
CodeSearchNetCorpus (config=None, provide=None, share_dependency_objects=False, build=True) |
the module base class |
CodeSearchNetChallenge (config=None, provide=None, share_dependency_objects=False, build=True) |
CodeSearchNetChallenge can only be used for training but not for evaluation since qrels is not provided |
COVID (config=None, provide=None, share_dependency_objects=False, build=True) |
Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit |
CovidQA (config=None, provide=None, share_dependency_objects=False, build=True) |
the module base class |
-
class
capreolus.benchmark.
Benchmark
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
profane.ModuleBase
the module base class
-
class
capreolus.benchmark.
DummyBenchmark
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
the module base class
-
class
capreolus.benchmark.
WSDM20Demo
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
Robust04 benchmark equivalent to robust04.yang19
-
class
capreolus.benchmark.
Robust04Yang19
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
Robust04 benchmark using the folds from Yang et al. [1]
[1] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. SIGIR 2019.
-
class
capreolus.benchmark.
ANTIQUE
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
A Non-factoid Question Answering Benchmark from Hashemi et al. [1]
[1] Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W. Bruce Croft. 2020. ANTIQUE: A non-factoid question answering benchmark. ECIR 2020.
-
class
capreolus.benchmark.
MSMarcoPassage
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
the module base class
-
class
capreolus.benchmark.
CodeSearchNetCorpus
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
the module base class
-
class
capreolus.benchmark.
CodeSearchNetChallenge
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
CodeSearchNetChallenge can only be used for training but not for evaluation since qrels is not provided
-
class
capreolus.benchmark.
COVID
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit
-
class
capreolus.benchmark.
CovidQA
(config=None, provide=None, share_dependency_objects=False, build=True)[source]¶ Bases:
capreolus.benchmark.Benchmark
the module base class