Question

A research team shares a Slurm-managed cluster where each node has 8 GPUs. Users report that GPU-heavy jobs sometimes land on nodes that are already fully utilized, causing failures, while other nodes sit idle. Investigation shows job scripts request nodes and CPU cores but the scheduler is not tracking GPUs as an allocatable resource. What is the most appropriate fix to ensure Slurm schedules jobs based on GPU availability?

Accepted Answer

Configure GPUs as a Generic Resource (GRES) in slurm.conf and gres.conf, and have users request GPUs with --gres=gpu:N

Answer

Instruct users to add more --cpus-per-task so each job reserves an entire node

Answer

Increase the Slurm scheduler polling interval so it re-evaluates node load more frequently

Answer

Enable node oversubscription so multiple jobs can share the same GPUs