A python interface to TREC's ndeval.c, used for computing diversity retrieval metrics.
From pip:
pip install pyndevalOr install from source:
git clone https://github.com/seanmacavaney/pyndeval.git
cd pyndeval
python setup.py installimport pyndeval
from pyndeval import SubtopicQrel, ScoredDoc
# provide qrels as a list of tuples
qrels = [
SubtopicQrel("0", "a", "A", 1),
SubtopicQrel("0", "b", "B", 1),
SubtopicQrel("0", "b", "D", 1),
SubtopicQrel("0", "c", "C", 1),
]
# provide run as a list of tuples
run = [
ScoredDoc("0", "A", 9.3),
ScoredDoc("0", "D", 8.4),
ScoredDoc("0", "E", 8.1), # not in qrels
ScoredDoc("0", "B", 7.6),
# C not retrieved
]
pyndeval.ndeval(qrels, run)
{'0': {
'ERR-IA@5': 0.3933,
'ERR-IA@10': 0.3907,
'ERR-IA@20': 0.3907,
'nERR-IA@5': 0.8297,
'nERR-IA@10': 0.8297,
'nERR-IA@20': 0.8297,
'alpha-DCG@5': 0.4052,
'alpha-DCG@10': 0.3998,
'alpha-DCG@20': 0.3997,
'alpha-nDCG@5': 0.7868,
'alpha-nDCG@10': 0.7868,
'alpha-nDCG@20': 0.7868,
'NRBP': 0.3906,
'nNRBP': 0.8620,
'MAP-IA': 0.5000,
'P-IA@5': 0.2000,
'P-IA@10': 0.1000,
'P-IA@20': 0.0500,
'strec@5': 0.6666,
'strec@10': 0.6666,
'strec@20': 0.6666
}}pyndeval supports the following measures:
ERR-IA@knERR-IA@kalpha-DCG@kalpha-nDCG@kP-IA@kstrec@kNRBPnNRBPMAP-IA
Measures with @k support values from 1-20 (upper limit from ndeval.c).
Measures are provided as a list of strings with the measures= parameter.
pyndeval.ndeval(qrels, run, measures=["ERR-IA@7", "MAP-IA"])If you're running multiple times for the same set of qrels and measures, you can speed it up by
building a RelevanceEvaluator object, which caches the internal qrel representations.
ev = pyndeval.RelevanceEvaluator(qrels)
ev.evaluate(run1)
ev.evaluate(run2)
ev.evaluate(run3)
ev.evaluate(run4)for result in pyndeval.ndeval_iter(qrels, run):
{"query_id": "0", ...}
for result in ev.evaluate_iter(run):
{"query_id": "0", ...}