Question

During a large multi-node training run, one node repeatedly crashes its training process. The cluster operator wants to determine whether a specific GPU is experiencing hardware faults such as ECC errors or falling off the bus. Which NVIDIA tool and data source should the operator use to programmatically collect GPU health events, XID error codes, and run active diagnostics before deciding whether to drain the node?

Accepted Answer

NVIDIA DCGM, which exposes health checks, XID error reporting, and diagnostic levels (dcgmi diag) for GPU fault detection

Answer

Kubernetes kubectl describe node, which reports GPU hardware fault codes and ECC error counts directly

Answer

The CUDA Toolkit's nvcc compiler diagnostics, which flag GPU hardware errors at build time

Answer

Slurm's sacct accounting database, which records per-GPU XID codes and thermal fault events