Question

A platform engineer manages a Kubernetes cluster using the NVIDIA GPU Operator. After a rolling worker-node OS upgrade, GPU-requesting pods on the upgraded nodes stay in 'Pending' state, and `kubectl describe node` shows the nodes no longer advertise the `nvidia.com/gpu` extended resource. The nodes themselves are Ready, and other (non-GPU) pods schedule normally. What is the most likely root cause to investigate first?

Accepted Answer

The NVIDIA device plugin DaemonSet pod on the upgraded nodes failed to become ready because the GPU driver could not build or load against the new kernel version

Answer

The Kubernetes scheduler's fair-share policy is deprioritizing GPU pods after the upgrade

Answer

DCGM Exporter stopped scraping metrics, which removed the GPU resource advertisement from the node

Answer

The pods are missing a matching GPU node toleration, so the scheduler rejects them