Question

A Kubernetes cluster has several A100 GPU nodes. One node has been partitioned so a single GPU exposes both a 3g.20gb slice and two 1g.5gb slices to serve differently sized inference models on the same physical GPU. Pods requesting the smaller MIG profile remain Pending on this node even though free 1g.5gb slices exist. Which cluster configuration is the most likely cause?

Accepted Answer

The GPU Operator's MIG strategy is set to 'single', which requires all MIG devices on a node to be the same profile and cannot advertise heterogeneous slices

Answer

MIG mode is disabled on the GPU, so no slices can be scheduled at all

Answer

The node lacks the NVIDIA device plugin, so no GPU resources are advertised to the scheduler

Answer

The pods are missing a nodeSelector for the GPU product label, so they were scheduled elsewhere