Question

A cluster administrator needs to perform preventive maintenance on a Slurm compute node that is currently running several multi-day training jobs. The administrator wants to prevent any new jobs from being scheduled onto the node while allowing the currently running jobs to finish naturally before the node is taken offline. Which Slurm action best achieves this goal?

Accepted Answer

Set the node to the DRAIN state using scontrol so no new jobs are allocated while existing jobs continue running

Answer

Set the node to the DOWN state using scontrol to immediately remove it from the cluster

Answer

Cancel all running jobs with scancel and then reboot the node

Answer

Lower the node's fairshare priority so the scheduler stops selecting it