Question

A cluster operator notices that several Slurm training jobs intermittently fail minutes after starting, always landing on the same few GPU nodes. The failures do not appear during normal idle periods. The operator wants to automatically prevent jobs from being scheduled onto GPUs that have degraded or failing hardware before the job begins. Which approach best addresses this?

Accepted Answer

Integrate DCGM diagnostics (dcgm health checks) as a prolog step so nodes with unhealthy GPUs are drained or excluded before jobs run

Answer

Increase the Slurm job time limit so failing jobs have more time to complete before timing out

Answer

Reduce the per-job GPU memory request so workloads fit into a smaller partition

Answer

Enable MIG on all affected nodes to isolate tenants from one another