Question

Your team is training a large image classification CNN on a single NVIDIA A100 GPU using TensorFlow. To fit a larger batch size and speed up training, you enabled mixed-precision (float16) training. After the change, throughput improved significantly, but the loss frequently becomes NaN early in training and the model fails to converge. What is the most appropriate action to fix the instability while keeping the performance benefits of mixed precision?

Accepted Answer

Apply dynamic loss scaling so small gradients are not lost to float16 underflow while keeping numerically sensitive operations in float32.

Answer

Revert entirely to float32 training since mixed precision is inherently unstable on A100 GPUs.

Answer

Reduce the learning rate to near zero to prevent the loss from ever spiking to NaN.

Answer

Switch the hardware to a TPU v3 pod, which does not experience float16 numerical issues.