Question

A cluster administrator receives complaints that GPU nodes appear busy but training throughput is low. They want to attribute measured GPU SM activity, memory usage, and energy to individual Slurm jobs so they can identify which jobs underutilize their allocated GPUs. Which approach provides per-job GPU accounting metrics with the least custom development?

Accepted Answer

Enable DCGM job statistics (dcgmi stats / job-level accounting) integrated with the Slurm prolog/epilog to record per-job GPU utilization, memory, and power

Answer

Parse the output of nvidia-smi at 1-second intervals via a cron job and manually correlate timestamps to Slurm job IDs afterward

Answer

Read the GPU fan speed and clock registers with nvidia-settings for each node and estimate utilization from thermal behavior

Answer

Rely on the Slurm sacct CPU efficiency report (seff) to infer GPU utilization from CPU time consumed by each job