🔥 News
rank_eval will be featured in ECIR 2022, the 44th European Conference on Information Retrieval!
🤖 Dev Bulletin
We are aware rank_eval is currently not working on Google Colab.
That's because Google Colab runs Python 3.6 while rank_eval currently requires a newer Python version.
I will try to downgrade the required Python version soon.
If you experienced the numba.typed
issue, it should now be solved. Please, re-install rank_eval.
⚡️ Introduction
rank_eval is a library of fast ranking evaluation metrics implemented in Python, leveraging Numba for high-speed vector operations and automatic parallelization.
It allows you to compare different runs, perform statistical tests, and export a LaTeX table for your scientific publications.
We strongly incourage you to check the example folder to learn how to use rank_eval in just a few minutes.
✨ Available Metrics
- Hits
- Precision
- Recall
- rPrecision
- Mean Reciprocal Rank (MRR)
- Mean Average Precision (MAP)
- Normalized Discounted Cumulative Gain (NDCG)
The metrics have been tested against TREC Eval for correctness.
🔌 Installation
pip install rank_eval
💡 Usage
Create Qrels and Run
from rank_eval import Qrels, Run, evaluate
qrels = Qrels()
qrels.add_multi(
q_ids=["q_1", "q_2"],
doc_ids=[
["doc_12", "doc_25"],
["doc_11", "doc_2"],
],
scores=[
[5, 3],
[6, 1],
],
)
run = Run()
run.add_multi(
q_ids=["q_1", "q_2"],
doc_ids=[
["doc_12", "doc_23", "doc_25", "doc_36", "doc_32", "doc_35"],
["doc_12", "doc_11", "doc_25", "doc_36", "doc_2", "doc_35"],
],
scores=[
[0.9, 0.8, 0.7, 0.6, 0.5, 0.4],
[0.9, 0.8, 0.7, 0.6, 0.5, 0.4],
],
)
Evaluate
evaluate(qrels, run, "ndcg@5")
>>> 0.7861
evaluate(qrels, run, ["map@5", "mrr"])
>>> {"map@5": 0.6416, "mrr": 0.75}
run.mean_scores
>>> {"ndcg@5": 0.7861, "map@5": 0.6416, "mrr": 0.75}
dict(run.scores)
>>> {"ndcg@5": {"q_1": 0.9430, "q_2": 0.6292},
"map@5": {"q_1": 0.8333, "q_2": 0.4500},
"mrr": {"q_1": 1.0000, "q_2": 0.5000}}
Compare
report = compare(
qrels=qrels,
runs=[run_1, run_2, run_3, run_4, run_5],
metrics=["map@100", "mrr@100", "ndcg@10"],
max_p=0.01
)
print(report)
Output:
# Model MAP@100 MRR@100 NDCG@10
--- ------- ---------- ---------- ----------
a model_1 0.3202ᵇ 0.3207ᵇ 0.3684ᵇᶜ
b model_2 0.2332 0.2339 0.239
c model_3 0.3082ᵇ 0.3089ᵇ 0.3295ᵇ
d model_4 0.3664ᵃᵇᶜ 0.3668ᵃᵇᶜ 0.4078ᵃᵇᶜ
e model_5 0.4053ᵃᵇᶜᵈ 0.4061ᵃᵇᶜᵈ 0.4512ᵃᵇᶜᵈ
📖 Examples
📚 Documentation
To be updated! Please, refer to the examples in the meantime.
Search the documentation for more details and examples.
🎓 Citation
If you use rank_eval to evaluate results for your scientific publication, please consider citing it:
@misc{rankEval2021,
title = {Rank\_eval: Blazing Fast Ranking Evaluation Metrics in Python},
author = {Bassani, Elias},
year = {2021},
publisher = {GitHub},
howpublished = {\url{https://github.com/AmenRa/rank_eval}},
}
🎁 Feature Requests
Would you like to see a new metric implemented? Please, open a new issue.
🤘 Want to contribute?
Would you like to contribute? Please, drop me an e-mail.
📄 License
rank_eval is an open-sourced software licensed under the MIT license.