{"repo":"OrderLab/TrainCheck","free":true,"listed":false,"github":"https://github.com/OrderLab/TrainCheck","clone":"git clone https://github.com/OrderLab/TrainCheck.git","description":"An Observability Framework for AI Training","language":"Python","stars":73,"topics":["debugging","machine-learning","pytorch","observability"],"license":null,"category":"machine-learning","readme_excerpt":"TrainCheck: Invariant Checking for AI Training TrainCheck catches silent training bugs by learning what a healthy run does, then checking a new run against those learned invariants. It works by tracing PyTorch API calls and model state changes, so you can inspect training behavior before a loss curve or final metric tells you something went wrong. Install Install TrainCheck in the same Python environment that runs your training script: For CUDA, conda, and source-install details, see the Installation Guide. Use TrainCheck TrainCheck has four main steps. 1. Collect a Reference Trace Run traincheck-collect on a known-good training script. This should be a short run that covers the training behavior you want TrainCheck to learn. 2. Infer Invariants Turn the reference trace into invariants: 3. Collect a Target Trace Run the target training script with the inferred invariants. Passing --invariants lets TrainCheck trace only the APIs and variables needed for those checks. For long target runs, trace fewer steps: 4. Check the Target Run For live checking, start traincheck-onlinecheck while the target run is writing traces: The easier offline path is to wait for trace collection to finish, then run: Both checkers write a results directory with failure logs and a report.html summary. Learn More - Use TrainCheck explains the full workflow and output files. - 5-Minute Tutorial walks through a real silent training issue. - Installation Guide covers environment setup. - Technical Document","default_branch":null,"files":null,"tree":[],"storefront":"/r/OrderLab","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/OrderLab/TrainCheck/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}