Question

Your team is training a large vision transformer on a single NVIDIA A100 (40 GB) GPU using PyTorch on Vertex AI. Research shows the model converges best with an effective batch size of 512, but the GPU runs out of memory at a batch size of 64. You must not change hardware and want to preserve the target effective batch size without degrading convergence. What is the most appropriate technique?

Accepted Answer

Enable gradient accumulation over 8 micro-batches of size 64 before applying an optimizer step

Answer

Reduce the batch size to 64 and proportionally lower the learning rate to compensate

Answer

Switch to a MultiWorkerMirroredStrategy across additional GPUs to split the batch

Answer

Cast all model weights to int8 quantization to reduce memory footprint during training