Question

A researcher submits a Slurm batch job requesting 4 GPUs on a cluster where each node has 8 GPUs and several nodes appear idle. The job remains in the PD (pending) state with reason code 'QOSMaxGRESPerUser'. What is the most likely cause?

Accepted Answer

The job's Quality of Service (QOS) policy caps the total number of GPUs the user may allocate simultaneously, and the user has reached that limit

Answer

The idle nodes lack the NVIDIA driver, so Slurm cannot schedule GPU jobs on them

Answer

The job requested more GPUs than exist on any single node in the cluster

Answer

DCGM health checks flagged the idle GPUs as unhealthy and drained the nodes