Question

Your team is training a deep neural network for image classification on Vertex AI using 8 GPUs with data parallelism. To improve throughput, you increased the global batch size from 256 to 4096 and proportionally scaled the base learning rate. However, training now diverges within the first few hundred steps, with the loss quickly becoming NaN. The single-GPU baseline with batch size 256 trained stably. What is the most appropriate change to stabilize training while retaining the large batch size?

Accepted Answer

Apply a learning rate warmup that gradually ramps the learning rate from a small value to the scaled target over the first several epochs

Answer

Revert to the single-GPU configuration with batch size 256 to avoid instability entirely

Answer

Switch from data-parallel to model-parallel distribution across the 8 GPUs

Answer

Reduce the number of GPUs to 2 so the effective batch size is smaller