capreolus.benchmark.covid

Module Contents

Classes

COVID(config=None, provide=None, share_dependency_objects=False, build=True) Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit that uses documents from CORD, the COVID-19 Open Research Dataset (https://www.semanticscholar.org/cord19).
CovidQA(config=None, provide=None, share_dependency_objects=False, build=True) Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.
capreolus.benchmark.covid.logger[source]
capreolus.benchmark.covid.PACKAGE_PATH[source]
class capreolus.benchmark.covid.COVID(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

Ongoing TREC-COVID bechmark from https://ir.nist.gov/covidSubmit that uses documents from CORD, the COVID-19 Open Research Dataset (https://www.semanticscholar.org/cord19).

module_name = covid[source]
dependencies[source]
data_dir[source]
topic_url = https://ir.nist.gov/covidSubmit/data/topics-rnd%d.xml[source]
qrel_url = https://ir.nist.gov/covidSubmit/data/qrels-rnd%d.txt[source]
lastest_round = 3[source]
config_spec[source]
build(self)[source]
download_if_missing(self)[source]
xml2trectopic(self, xmlfile)[source]
class capreolus.benchmark.covid.CovidQA(config=None, provide=None, share_dependency_objects=False, build=True)[source]

Bases: capreolus.benchmark.Benchmark

Base class for Benchmark modules. The purpose of a Benchmark is to provide the data needed to run an experiment, such as queries, folds, and relevance judgments.

Modules should provide:
  • a topics dict mapping query ids (qids) to queries
  • a qrels dict mapping qids to docids and relevance labels
  • a folds dict mapping a fold name to training, dev (validation), and testing qids
  • if these can be loaded from files in standard formats, they can be specified by setting the topic_file, qrel_file, and fold_file, respectively, rather than by setting the above attributes directly
module_name = covidqa[source]
dependencies[source]
url = https://raw.githubusercontent.com/castorini/pygaggle/master/data/kaggle-lit-review-%s.json[source]
available_versions = ['0.1', '0.2'][source]
datadir[source]
config_spec[source]
build(self)[source]
download_if_missing(self)[source]