{"repo":"facebookresearch/gcm","free":true,"listed":false,"github":"https://github.com/facebookresearch/gcm","clone":"git clone https://github.com/facebookresearch/gcm.git","description":"GPU Cluster Monitoring (GCM): Large-Scale AI Research Cluster Monitoring","language":"Python","stars":235,"topics":["ai","ai-cluster","ai-training","dcgm","health-checks","hpc","monitoring","nvml","slurm"],"license":"MIT","category":"analytics","readme_excerpt":"Meta GPU Cluster Monitoring (GCM) GCM is a set of tools used to do at-scale monitoring for HPC (High-Performance Computing) clusters, it powers Meta FAIR (Fundamental AI Research) AI workloads across over hundreds of thousands of GPUs at Meta. GCM is a monorepo with the following components: - Monitoring: Collects cluster statistics from the Slurm workload scheduler, providing visibility into job performance and resource utilization. - Health Checks: Verifies the proper functioning of hardware, software, network, storage, and services throughout the job lifecycle. - Telemetry Processor / GPU Metrics: Enhances OpenTelemetry data by correlating telemetry with Slurm metadata, enabling attribution of metrics (e.g., GPU utilization) to specific jobs and users. For more information, check our documentation. Join our Discord community! Contributing Each component has its own README with detailed guides: - Monitoring - Health Checks - slurmprocessor - shelper Possible Expansions - Integration with more GPU types (AMD, Intel, Custom Accelerators) - Support for additional schedulers beyond Slurm - Additional Slurm related Monitoring - Support for new exporters - Adding support for Slurm REST API querying - Adding support for new Health Checks Code of Conduct Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the full text so that you can understand what actions will and will not be tolerated. The Team GPU Cluster Monitoring is actively ","default_branch":null,"files":null,"tree":[],"storefront":"/r/facebookresearch","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/facebookresearch/gcm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}