dagster-slurm
dagster-slurm links Dagster assets with Slurm cluster computing, while allowing assets to run locally during development.
What dagster-slurm is
dagster-slurm links Dagster assets with Slurm cluster computing, while allowing assets to run locally during development. In production, it transfers the current checkout payload and Pixi-locked environment, submits Slurm jobs, then returns their status and logs to Dagster. The default Slurm mode runs one job per asset, and Dagster can show Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours. Asset dependencies and lineage can connect ingestion, cluster computation, and downstream aggregation or publishing. Stable execution modes are local and Slurm. A Ray launcher supports distributed execution, GPU scheduling, and optional reuse of allocations within a run; heterogeneous Slurm jobs are experimental, while allocation reuse currently focuses on Ray assets. Spark integration is marked as work in progress. Cluster access uses SSH, with key authentication, jump hosts, and configurable host-key checking. Production expects a pre-deployed Pixi environment bundle on the cluster. The free Open-source software plan costs nothing and requires Dagster and Slurm; cluster execution requires a Slurm cluster. The project follows Python 3.12 development conventions and offers a Docker Compose Slurm stack for cluster workflow testing.
Good
- Assets can run locally or on Slurm.
- Returns Slurm status and logs to Dagster.
- Displays CPU, memory, elapsed time, and node-hours.
- SSH supports keys, jump hosts, and host-key settings.
Watch out
- Cluster execution requires a Slurm cluster.
- Production requires a pre-deployed environment bundle.
- Spark integration is work in progress.
- Heterogeneous Slurm jobs are experimental.
Who it suits
It is aimed at Dagster users developing locally and running compute-heavy assets, including model training and hyper-parameter sweeps, on Slurm clusters. It also suits teams that want Dagster to display job state and resource details.
Verdict
dagster-slurm connects Dagster’s asset workflow to local and Slurm execution, with job details available in Dagster. Ray support is available, but Spark and heterogeneous jobs have stated limits.
Plans and prices
From dagster-slurm's own pricing page, checked 5 Oct 2026.
| Plan | Price | What it gives you |
|---|---|---|
| Open-source software | Free | Dagster and Slurm required · Python 3.12 development conventions · Slurm cluster needed for cluster execution |
The facts
Each one read from dagster-slurm's own pages, with the date it was checked.
What it does
- Cluster connectivity
- The product connects to Slurm clusters over SSH and supports key-based authentication, jump hosts, and configurable host-key checking.checked 5 Oct 2026
- Environment packaging
- Jobs use Pixi environments, and production mode expects a pre-deployed environment bundle on the cluster.checked 5 Oct 2026
- Execution modes
- The stable execution modes are local and Slurm, with one Slurm job per asset by default.checked 5 Oct 2026
- Experimental features
- Heterogeneous Slurm jobs are experimental; run-scoped allocation reuse is currently focused on Ray assets.checked 5 Oct 2026
- Intended users
- The docs describe laptop-first development and cluster use for compute-heavy Dagster assets, including model training and hyper-parameter sweeps.checked 5 Oct 2026
- Job submission
- It transfers the current checkout payload and Pixi-locked environment, submits jobs with Slurm, and returns job status and logs to Dagster.checked 5 Oct 2026
- Local development
- Assets can run locally without SSH or Slurm, and the project provides a Docker Compose Slurm stack for testing cluster workflows.checked 5 Oct 2026
- Observability
- Dagster can display Slurm state, logs, CPU efficiency, memory, elapsed time, and node-hours for asset runs.checked 5 Oct 2026
- Pipeline orchestration
- Dagster asset dependencies and lineage can connect data ingestion, Slurm compute, and downstream aggregation or publishing.checked 5 Oct 2026
- Purpose
- dagster-slurm lets Dagster assets run locally during development and on a Slurm cluster in production.checked 5 Oct 2026
- Ray integration
- Ray is a first-class launcher for distributed execution, GPU scheduling, and optional run-scoped allocation reuse.checked 5 Oct 2026
- Secrets handling
- Cluster connection details can be provided through runtime environment variables so credentials can be kept out of the repository.checked 5 Oct 2026
- Spark integration
- The documentation labels the Spark integration as work in progress.checked 5 Oct 2026
- Support
- The project directs users to GitHub Discussions for questions and how-to requests, GitHub Issues for bugs and feature requests, and Dagster Slack for broader ecosystem questions.checked 5 Oct 2026
Alternatives to dagster-slurm
- 1
Flowable
Free planOpen sourceAndroidiOSWebAPI8.7From $99 a month - 2
Inngest
Free planOpen sourceWebAPILinuxmacOS8.3From $99 a month - 3
Windmill
Free planOpen sourceWebAPILinuxSelf-hosted8.3From $120 a month - 4
Flyte
Free planOpen sourceWebAPILinuxSelf-hosted8.0Free plan - 5
Kestra
Free planOpen sourceWebAPILinuxmacOS7.9Free plan - 6
ActiveBatch
AndroidiOSWebAPI7.6$10,000 one-time