garrus

In the middle of some calibrations...

Garrus is a python framework for better confidence estimate of deep neural networks. Modern networks are overconfident estimators, that makes themselves unreliable and therefore limits the deployment of them in safety-critical applications.

Garrus provides tools for high quality confidence estimation such as confidence calibration and ordinal ranking methods, helping networks to know correctly what they do not know.

---

Installation:

pip install -U garrus

Documentation:

• master v0.2.0
• develop v0.2.0

Roadmap:

• Core:
  • Calibration metrics:
    • ECE
    • NLL
    • Brier
  • Ordinal Ranking Metrics:
    • AURC
    • E-AURC
    • AUPRE
    • FPR-n%-TPR
  • Visualizations:
    • Reliability Diagram
    • Confidence Histogram
  • Garrus Profiling
• Confidence Calibration:
  • Scaling:
    • Platt
    • Temperature
  • Binning:
    • Histogram
    • Isotonic Regression
• Confidence Regularization:
  • Losses:
    • Correctness Ranking Loss
    • Focal Entropy Penalized Loss
  • Language Model Beam Search
• Confidence Networks:
  • ConfidNet
  • GarrusNet

---

Citation:

Please use this bibtex if you want to cite this repository in your publications:

@misc{garrus,
    author = {Kalashnikov, Alexander},
    title = {Deep neural networks calibration framework},
    year = {2021},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/sleep3r/garrus}},
}

References:

Papers
[1] Guo, Chuan, et al. "On calibration of modern neural networks." International Conference on Machine Learning. PMLR, 2017. APA
[2] Moon, Jooyoung, et al. "Confidence-aware learning for deep neural networks." international conference on machine learning. PMLR, 2020.
[3] Kumar, Ananya, Percy Liang, and Tengyu Ma. "Verified uncertainty calibration." arXiv preprint arXiv:1909.10155 (2019).