capreolus.eval.msmarco_eval

This module computes evaluation metrics for MSMARCO dataset on the ranking task.

Creation Date : 06/12/2018 Last Modified : 1/21/2019 Authors : Daniel Campos <dacamp@microsoft.com>, Rutger van Haasteren <ruvanh@microsoft.com>

Module Contents

Functions

qrel_trec_to_msmarco(qrels)
runs_trec_to_msmarco(runs)
quality_checks_qids(qids_to_relevant_passageids, ...)	Perform quality checks on the dictionaries
compute_metrics(qids_to_relevant_passageids, ...)	Compute MRR metric
compute_metrics_from_files(trec_qrels, trec_runs[, ...])
main()	Command line:

Attributes

MaxMRRRank

capreolus.eval.msmarco_eval.MaxMRRRank = 10

capreolus.eval.msmarco_eval.qrel_trec_to_msmarco(qrels)

capreolus.eval.msmarco_eval.runs_trec_to_msmarco(runs)

capreolus.eval.msmarco_eval.quality_checks_qids(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)

Perform quality checks on the dictionaries

Args: p_qids_to_relevant_passageids (dict): dictionary of query-passage mapping

Dict as read in with load_reference or load_reference_from_stream

p_qids_to_ranked_candidate_passages (dict): dictionary of query-passage candidates :returns: Boolean whether allowed, message to be shown in case of a problem :rtype: bool,str

capreolus.eval.msmarco_eval.compute_metrics(qids_to_relevant_passageids, qids_to_ranked_candidate_passages)

Compute MRR metric Args: p_qids_to_relevant_passageids (dict): dictionary of query-passage mapping

Dict as read in with load_reference or load_reference_from_stream

p_qids_to_ranked_candidate_passages (dict): dictionary of query-passage candidates :returns: dictionary of metrics {‘MRR’: <MRR Score>} :rtype: dict

capreolus.eval.msmarco_eval.compute_metrics_from_files(trec_qrels, trec_runs, perform_checks=True)

capreolus.eval.msmarco_eval.main()

Command line: python msmarco_eval_ranking.py <path_to_reference_file> <path_to_candidate_file>