Question

A platform team dedicates a group of GPU nodes and taints them with 'gpu=true:NoSchedule'. A data science team then deploys a training Pod that includes a matching toleration for that taint. However, the training Pod ends up scheduled onto ordinary CPU nodes instead of the GPU nodes. Why did this happen?

Accepted Answer

A toleration only allows a Pod to be placed on tainted nodes; it does not attract or force the Pod onto them, so nodeAffinity or a nodeSelector is still needed to target the GPU nodes.

Answer

A NoSchedule taint blocks all Pods regardless of tolerations, so the Pod could never land on the GPU nodes.

Answer

The Pod must also declare a GPU resource request, which is the only mechanism the scheduler uses to place Pods on tainted nodes.

Answer

Tolerations are evaluated only by the kubelet after scheduling, so they have no effect on where the scheduler places the Pod.