Question

Your team has a TensorFlow 2 image classification model that currently trains on a single GPU and takes 18 hours per run. You provisioned a Vertex AI custom training job on one machine equipped with 4 NVIDIA A100 GPUs and want to reduce training time with minimal code changes while keeping the model architecture unchanged. Which approach should you use to distribute training across the 4 GPUs?

Accepted Answer

Wrap the model construction and compilation inside tf.distribute.MirroredStrategy scope and let it synchronize gradients across the 4 GPUs

Answer

Use tf.distribute.MultiWorkerMirroredStrategy and configure a TF_CONFIG environment variable listing the 4 GPUs as separate workers

Answer

Use tf.distribute.ParameterServerStrategy with one GPU acting as the parameter server and the remaining three as workers

Answer

Manually split each training batch into four shards and run four independent training processes, then average the resulting model weights after each epoch