{"repo":"nshinde/stallscope","free":true,"listed":false,"github":"https://github.com/nshinde/stallscope","clone":"git clone https://github.com/nshinde/stallscope.git","description":"Early-warning bottleneck profiler for GPU training nodes: GPU + RDMA fabric telemetry with job-level classification","language":"Python","stars":10,"topics":["gpu-monitoring","mlops","nccl","rdma","rocev2","slurm","prometheus"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"stallscope Correlate GPU training job health with RDMA fabric counters on the same node with shared job context. stallscope is a Python 3.10+ observability utility for GPU training nodes. It collects local GPU, host, network, RDMA/RoCEv2, PFC pause, and scheduler context signals, then applies rule-based profiling to label the current job as FAST , SLOW , FAIL RISK , or UNKNOWN . The implementation is stdlib-only at runtime and is designed to keep parsing testable without GPU, RDMA, or Slurm/Kubernetes hardware by using pure parser functions and fixture-backed tests. Quickstart Run continuously every 30 seconds: Write Prometheus textfile metrics for node exporter: Send alert webhook payloads when alerts are active: Run the optional NCCL all-reduce benchmark if all reduce perf is installed and available in PATH : What It Collects Metric family Source What a spike means --- --- --- GPU utilization, memory, temperature, power, throttle reasons nvidia-smi --query-gpu=... Low utilization can indicate stalls; high memory use, thermal risk, or throttling can contribute to slow or risky jobs. Interface bytes, packets, errors, drops /proc/net/dev Errors and drops indicate local network interface health problems visible to the host. Host load and memory availability os.getloadavg() , /proc/meminfo High CPU load or low available memory can indicate host-side pressure. RDMA port traffic and basic link counters /sys/class/infiniband/ /ports/ /counters/port rcv data , port xmit data , port ","default_branch":null,"files":null,"tree":[],"storefront":"/r/nshinde","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/nshinde/stallscope/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}