Slurm GPU clusters move to Slurm-on-Kubernetes stack with self-healing workers

All newly provisioned Slurm GPU clusters now run on a new Slurm-on-Kubernetes stack with significant reliability improvements. Existing clusters can be migrated in place.

What's new:

• Self-healing worker daemons: The Slurm worker daemon is now supervised and auto-restarts on crash, so transient failures recover without operator intervention or impact on healthy nodes.
• Durable job accounting: Job history (sacct) is now persisted on durable, PVC-backed storage. Restarts and pod reschedules no longer wipe accounting data.
• Correct process tracking and cleanup: Job processes (including daemonized children) are tracked at the kernel cgroup level and reliably cleaned up at job completion. No more orphaned processes holding GPU memory or /dev/shm.
• Zombie reaping: A dedicated init process reaps orphaned children, preventing PID-table exhaustion from blocking new jobs.
• GPU state correctness: The Slurm GPU view is rebuilt fresh on every node start, eliminating "GPU not found" failures after pod reschedules.
• Per-cluster GPU utilization metrics: DCGM metrics are now exposed in your cluster's Grafana dashboards for fine-grained utilization visibility.

See Slurm configuration for more details.