Question

Your team is training a 40-billion-parameter transformer language model from scratch. Each model replica's parameters, optimizer states, and activations far exceed the 40 GB of memory available on a single GPU, so the full model cannot fit on one device. You have access to a multi-node cluster of GPUs on Vertex AI custom training. Which distributed training approach should you adopt to make training feasible?

Accepted Answer

Model/tensor parallelism (e.g., sharding layers and parameters across GPUs), optionally combined with data parallelism, so a single model instance spans multiple devices

Answer

Standard synchronous data parallelism, replicating the full model on each GPU and averaging gradients across replicas

Answer

Reduce the global batch size to 1 and use gradient accumulation so the model fits on a single GPU

Answer

Asynchronous parameter-server training with each worker holding a full model copy and pushing updates independently