capreolus.eval.msmarco_eval

This module computes evaluation metrics for MSMARCO dataset on the ranking task.

Creation Date : 06/12/2018 Last Modified : 1/21/2019 Authors : Daniel Campos <dacamp@microsoft.com>, Rutger van Haasteren <ruvanh@microsoft.com>

Module Contents

Functions

qrel_trec_to_msmarco(qrels)

runs_trec_to_msmarco(runs)

quality_checks_qids(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)

Perform quality checks on the dictionaries

compute_metrics(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)

Compute MRR metric

compute_metrics_from_files(trec_qrels, trec_runs, perform_checks=True)

main()

Command line:

Attributes

MaxMRRRank

capreolus.eval.msmarco_eval.MaxMRRRank = 10[source]
capreolus.eval.msmarco_eval.qrel_trec_to_msmarco(qrels)[source]
capreolus.eval.msmarco_eval.runs_trec_to_msmarco(runs)[source]
capreolus.eval.msmarco_eval.quality_checks_qids(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)[source]

Perform quality checks on the dictionaries

Args: p_qids_to_relevant_passageids (dict): dictionary of query-passage mapping

Dict as read in with load_reference or load_reference_from_stream

p_qids_to_ranked_candidate_passages (dict): dictionary of query-passage candidates :returns: Boolean whether allowed, message to be shown in case of a problem :rtype: bool,str

capreolus.eval.msmarco_eval.compute_metrics(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)[source]

Compute MRR metric Args: p_qids_to_relevant_passageids (dict): dictionary of query-passage mapping

Dict as read in with load_reference or load_reference_from_stream

p_qids_to_ranked_candidate_passages (dict): dictionary of query-passage candidates :returns: dictionary of metrics {‘MRR’: <MRR Score>} :rtype: dict

capreolus.eval.msmarco_eval.compute_metrics_from_files(trec_qrels, trec_runs, perform_checks=True)[source]
capreolus.eval.msmarco_eval.main()[source]

Command line: python msmarco_eval_ranking.py <path_to_reference_file> <path_to_candidate_file>