Security News
Research
Data Theft Repackaged: A Case Study in Malicious Wrapper Packages on npm
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Evalica [ɛˈʋalit͡sa] (eh-vah-lee-tsah) is a Python library that transforms pairwise comparisons into ranked lists of items. It offers convenient high-performant Rust implementations of the corresponding methods via PyO3, and additionally provides naïve Python code for most of them. Evalica is fully compatible with NumPy arrays and pandas data frames.
The logo was created using Recraft.
[!NOTE] The demonstration paper describing Evalica has been accepted at the COLING 2025 conference in Abu Dhabi!
Imagine that we would like to rank the different meals and have the following dataset of three comparisons produced by food experts.
Item X | Item Y | Winner |
---|---|---|
pizza | burger | x |
burger | sushi | y |
pizza | sushi | tie |
Given this hypothetical example, Evalica takes these three columns and computes the outcome of the given pairwise comparison according to the chosen model. Note that the first argument is the column Item X
, the second argument is the column Item Y
, and the third argument corresponds to the column Winner
.
>>> from evalica import elo, Winner
>>> result = elo(
... ['pizza', 'burger', 'pizza'],
... ['burger', 'sushi', 'sushi'],
... [Winner.X, Winner.Y, Winner.Draw],
... )
>>> result.scores
pizza 1014.972058
burger 970.647200
sushi 1014.380742
Name: elo, dtype: float64
As a result, we obtain Elo scores of our items. In this example, pizza
was the most favoured item, sushi
was the runner-up, and burger
was the least preferred item.
Item | Score |
---|---|
pizza | 1014.97 |
burger | 970.65 |
sushi | 1014.38 |
Evalica also provides a simple command-line interface, allowing the use of these methods in shell scripts and for prototyping.
$ evalica -i food.csv bradley-terry
item,score,rank
Tacos,2.509025136024378,1
Sushi,1.1011561298265815,2
Burger,0.8549063627182466,3
Pasta,0.7403814336665869,4
Pizza,0.5718366915548537,5
Refer to the food.csv file as an input example.
Evalica has a built-in Gradio application that can be launched as python3 -m evalica.gradio
. Please ensure that the library was installed as pip install evalica[gradio]
.
Method | In Python | In Rust |
---|---|---|
Counting | ✅ | ✅ |
Average Win Rate | ✅ | ✅ |
Bradley–Terry | ✅ | ✅ |
Elo | ✅ | ✅ |
Eigenvalue | ✅ | ✅ |
PageRank | ✅ | ✅ |
Newman | ✅ | ✅ |
Coming soon.
Copyright (c) 2024 Dmitry Ustalov. See LICENSE for details.
FAQs
Evalica, your favourite evaluation toolkit.
We found that evalica demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.