UNIQUE is a Python package for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
UNcertaInty QUantification bEnchmark: a Python library for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
UNIQUE provides methods for quantifying and evaluating the uncertainty of Machine Learning (ML) models predictions. The library allows to combine and benchmark multiple uncertainty quantification (UQ) methods simultaneously, generates intuitive visualizations, evaluates the goodness of the UQ methods against established metrics, and in general enables the users to get a comprehensive overview of their ML model's performances from an uncertainty quantification perspective.
UNIQUE is a model-agnostic tool, meaning that it does not depend on any specific ML model building platform or provides any ML model training functionality. It is lightweight, because it only requires the user to input their model's inputs and predictions.
High-level schema of 
UNIQUE's components.
UNIQUE is currently compatible with Python 3.8 through 3.12.1. To install the latest release and use the package as is, run the following in a compatible environment of choice:
pip install unique-uncertainty
or:
conda install -c conda-forge unique-uncertainty
# mamba install -c conda-forge unique-uncertainty
Check out the docs for more installation instructions.
Check out the docs for a complete set of instructions on how to prepare your data and the possible configurations offered by UNIQUE.
Finally, once the data and configuration files have been prepared, you can run UNIQUE in the following way:
from unique import Pipeline
# Prepare UNIQUE pipeline
pipeline = Pipeline.from_config("/path/to/config.yaml")
# Run UNIQUE pipeline
uq_methods_outputs, uq_evaluation_outputs = pipeline.fit()
# Returns: (Dict[str, np.ndarray], Dict[str, pd.DataFrame])
Fitting the Pipeline will return two dictionaries:
uq_methods_outputs: contains each UQ method's name (as in "UQ_Method_Name[Input_Name(s)]") and computed UQ values.uq_evaluation_outputs: contains, for each evaluation type (ranking-based, proper scoring rules, and calibration-based), the evaluation metrics outputs for all the corresponding UQ methods organized in pd.DataFrame.Additionally, UNIQUE also generates graphical outputs in the form of tables and evaluation plots (if display_outputs is enabled and the code is running in a JupyterNotebook cell).
For more hands-on examples and detailed usage, check out some of the examples in the docs.
Check out the docs for an in-depth overview of UNIQUE's concepts, functionalities, outputs, and references.
Any and all contributions and suggestions from the community are more than welcome and highly appreciated. If you wish to help us out in making UNIQUE even better, please check out our contributing guidelines.
Please note that we have a Code of Conduct in place to ensure a positive and inclusive community environment. By participating in this project, you agree to abide by its terms.
UNIQUE is licensed under the BSD 3-Clause License. See the LICENSE file.
If you find UNIQUE helpful for your work and/or research, please consider citing our work:
@misc{lanini2024unique,
title={UNIQUE: A Framework for Uncertainty Quantification Benchmarking},
author={Lanini, Jessica and Huynh, Minh Tam Davide and Scebba, Gaetano and Schneider, Nadine and Rodr{\'\i}guez-P{\'e}rez, Raquel},
year={2024},
doi={https://doi.org/10.26434/chemrxiv-2024-fmbgk},
}
For any questions or further details about the project, please get in touch with any of the following contacts:
FAQs
UNIQUE is a Python package for benchmarking uncertainty estimation and quantification methods for Machine Learning models predictions.
We found that unique-uncertainty demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.