{"repo":"ascii-supply-networks/dagster-slurm","free":true,"listed":false,"github":"https://github.com/ascii-supply-networks/dagster-slurm","clone":"git clone https://github.com/ascii-supply-networks/dagster-slurm.git","description":"Dagster SLURM integration","language":"Python","stars":59,"topics":["dagster","hpc","ray","slurm","spark","feray","metaxy"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"dagster-slurm --- Integration for running Dagster assets on Slurm HPC clusters. dagster-slurm lets you take the same Dagster assets from a laptop to a Slurm-backed supercomputer with minimal configuration changes. An European sovereign GPU cloud does not come out of nowhere maybe this project can support making HPC systems more accessible . See the docs for instructions. Status: Local execution and one-job-per-asset Slurm runs are production-ready today. Ray workloads can also opt into a run-scoped Slurm allocation when several compatible assets should share one Ray cluster. Heterogeneous jobs remain experimental. features Hydrating supercomputers with Dagster and Slurm to enhance developer experience. - Unified API : Same asset code works in dev (local) for rapid prototyping but just as well on a massive HPC system via SLURM. - Job launch : One Slurm submission per Dagster asset keeps behaviour predictable by default. Ray assets can explicitly opt into one Slurm allocation per Dagster run when the selected assets share one allocation shape. - Pluggable Launchers : Bash, Ray, Spark (WIP) - easy to add more - Environment Packaging : Automatic pixi -based environment packaging for remote execution via pixi-pack - Connection Pooling : SSH ControlMaster for efficient remote operations - Metrics Collection : Automatic collection of Slurm job metrics (CPU efficiency, memory, node-hours) - Production-Prepared : Proper error handling, cleanup, health checks 📊 Metrics Automatic metri","default_branch":null,"files":null,"tree":[],"storefront":"/r/ascii-supply-networks","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ascii-supply-networks/dagster-slurm/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}