{"repo":"probkit/probmetrics","free":true,"listed":false,"github":"https://github.com/probkit/probmetrics","clone":"git clone https://github.com/probkit/probmetrics.git","description":"Post-hoc calibration methods and metrics for classification","language":"Python","stars":72,"topics":["calibration","classification","machine-learning","metrics"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"Probmetrics: Classification metrics and post-hoc calibration This package (PyTorch-based) currently contains - post-hoc calibration methods, in particular: - a fast and accurate temperature scaling implementation described in [1] - an implementation of structured matrix scaling (SMS), a regularized version of matrix scaling introduced in [2] - implementations for all the post-hoc calibration methods in the CalArena benchmark [4] - classification metrics, especially metrics for assessing the quality of probabilistic predictions, in particular: - our classifier based $L p$ calibration error estimators from [3] It accompanies our papers [1] Rethinking Early Stopping: Refine, Then Calibrate (see also: vision experiments, tabular experiments, theory) [2] Structured Matrix Scaling for Multi-Class Calibration (see also: experiments) [3] A Variational Estimator for Lp Calibration Errors (see also all experiments) [4] CalArena: A Large-Scale Post-Hoc Calibration Benchmark (see also leaderboards and experiments) Please cite us if you use this repository for research purposes. Installation Probmetrics is available via To obtain all functionality, run pip install 'probmetrics[extra,dev,dirichletcal]' . - extra installs: - numba for SAGA based optimizers in logistic calibrators SVS, SMS and others. - relplot for smooth ECE (only works with scikit-learn versions Note: Over- and under-confidence metrics are designed for binary classification. For multi-class in a top-class fashion, please u","default_branch":null,"files":null,"tree":[],"storefront":"/r/probkit","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/probkit/probmetrics/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}