Question

Your team trains an image classification CNN on a single NVIDIA A100 GPU using a per-device batch size of 64 and a base learning rate that converges well. To reduce wall-clock time, you scale to 8 A100 GPUs on one node using data-parallel synchronous training with tf.distribute.MirroredStrategy, keeping the same per-device batch size of 64. After the change, training loss decreases much more slowly per epoch and final accuracy drops compared to the single-GPU run. What is the most appropriate first adjustment?

Accepted Answer

Increase the learning rate to account for the larger effective (global) batch size, for example by applying linear scaling relative to the batch-size increase.

Answer

Reduce the per-device batch size to 8 so the global batch size matches the original single-GPU batch size of 64.

Answer

Switch from MirroredStrategy to MultiWorkerMirroredStrategy to improve gradient synchronization across the GPUs.

Answer

Disable synchronous training and use asynchronous parameter-server updates to speed up convergence.