{"repo":"ElSacho/covmetrics","free":true,"listed":false,"github":"https://github.com/ElSacho/covmetrics","clone":"git clone https://github.com/ElSacho/covmetrics.git","description":"Conditional coverage metrics for set rules prediction.","language":"Python","stars":22,"topics":["conditional","conformal","coverage","diagnostics","metrics","prediction"],"license":"MIT","category":"analytics","readme_excerpt":"Covmetrics: conditional coverage metrics This package (PyTorch-based) currently contains different conditional coverage metrics, including our metric ERT (Excess risk of the target coverage). It accompanies our papers Conditional Coverage Diagnostics for Conformal Prediction. Please cite us if you use this repository for research purposes. Installation Covmetrics is available via Using conditional coverage metrics For a quick usage, you can evaluate a metric as follows: Where the object \"x\" is a feature vector of shape (n samples, n features) (numpy, torch or dataframe), and cover is a vector of shape (n samples,) with 0's or 1's The default classifier used to classify the outputs is a LightGBM classifier. You can change this by replacing the model class of the classifier. We recommand using our default model for a cheap computation, or better models like TabICLClassifier for instance. We provide an example with SGDClassifier, but would advise using better models. We recommend using our k-folds pre-implemented version to evaluate the conditional miscoverage by doing (default value is 5): But you can choose between training the classifier with some data and using it on other doing the following: Modifying the loss function The default loss used to evaluate the classifier provides a lower bound on the $L 1$-ERT. You can change the loss by doing : The package already provides several losses functions to evaluate your models. You can import them as follows: If you want to evaluat","default_branch":null,"files":null,"tree":[],"storefront":"/r/ElSacho","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ElSacho/covmetrics/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}