{"repo":"guilbaults/slurm-job-exporter","free":true,"listed":false,"github":"https://github.com/guilbaults/slurm-job-exporter","clone":"git clone https://github.com/guilbaults/slurm-job-exporter.git","description":"Prometheus exporter for the stats in the cgroup accounting with slurm. This will also collect stats of a job using NVIDIA GPUs.","language":"Python","stars":49,"topics":["prometheus-exporter","hpc-clusters","slurm"],"license":"Apache-2.0","category":"analytics","readme_excerpt":"Slurm-job-exporter Prometheus exporter for the stats in the cgroup accounting with Slurm. This will also collect stats of a job using NVIDIA GPUs. Requirements Slurm need to be configured with JobAcctGatherType=jobacct gather/cgroup . Stats are collected from the cgroups created by Slurm for each job. Python 3 with the following modules: prometheus client nvidia-ml-py (optional) DCGM is recommended when NVIDIA GPUs are installed: If DCGM is installed and running, it will be used instead of NVML. NVLINK and and a few other stats are only available with DCGM. MIG devices are supported. The proprietary package is required since this is using the profiling module of DCGM (datacenter-gpu-manager-4-proprietary) nvidia-smi -L is run in each cgroup to detect which GPU is allocated to a Slurm job. Usage Cgroup v1 vs v2 Slurm currently supports cgroup v1 and v2, but there are some limitations with v2, some metrics are not currently fully available on this exporter: memory.max usage in bytes becomes memory.peak , but this is not in any currently released kernel (torvalds/linux@8e20d4b). This is working on 5.14 kernel on EL9. cpuacct.usage percpu is exposed via eBPF in kernel 6.6+, but not through cgroupfs although it might be for a future kernel. At the moment, the exporter will get the overall CPU usage of the job and divide it by the number of allocated cores to get an average CPU usage per core. Sample","default_branch":null,"files":null,"tree":[],"storefront":"/r/guilbaults","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/guilbaults/slurm-job-exporter/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}