:mod:`capreolus.evaluator`
==========================

.. py:module:: capreolus.evaluator


Module Contents
---------------


Functions
~~~~~~~~~

.. autoapisummary::

   capreolus.evaluator.judged
   capreolus.evaluator.eval_runs
   capreolus.evaluator.eval_runfile
   capreolus.evaluator.search_best_run


.. data:: logger
   

   

.. data:: DEFAULT_METRICS
   :annotation: = ['P_1', 'P_5', 'P_10', 'P_20', 'judged_10', 'judged_20', 'judged_200', 'map', 'ndcg_cut_5', 'ndcg_cut_10', 'ndcg_cut_20', 'recall_100', 'recall_1000', 'recip_rank']

   

.. function:: judged(qrels, runs, n)


.. function:: eval_runs(runs, qrels, metrics, relevance_level)

   Evaluate runs loaded by Searcher.load_trec_run

   :param runs: a dict with format {qid: {docid: score}}, could be prepared by Searcher.load_trec_run
   :param qrels: dict, containing the judgements provided by benchmark
   :param metrics: str or list, metrics expected to calculate, e.g. ndcg_cut_20, etc

   :returns: score}, containing the evaluation score of specified metrics
   :rtype: a dict with format {metric


.. function:: eval_runfile(runfile, qrels, metrics, relevance_level)

   Evaluate a single runfile produced by ranker or reranker

   :param runfile: str, path to runfile
   :param qrels: dict, containing the judgements provided by benchmark
   :param metrics: str or list, metrics expected to calculate, e.g. ndcg_cut_20, etc

   :returns: score}, containing the evaluation score of specified metrics
   :rtype: a dict with format {metric


.. function:: search_best_run(runfile_dir, benchmark, primary_metric, metrics=None, folds=None)

   Select the runfile with respect to the specified metric

   :param runfile_dir: the directory path to all the runfiles to select from
   :param benchmark: Benchmark class
   :param primary_metric: str, metric used to select the best runfile , e.g. ndcg_cut_20, etc
   :param metrics: str or list, metric expected by be calculated on the best runs
   :param folds: str, the name of fold to select from

   :returns: a dict storing specified metric score and path to the corresponding runfile