Question

A machine learning engineer is training a large transformer model on a single SageMaker ml.g5.xlarge instance. Research suggests the model converges best with an effective batch size of 256, but the GPU runs out of memory at a batch size of 32. The engineer cannot afford a larger multi-GPU instance for this experiment. Which technique allows the engineer to achieve the desired effective batch size of 256 without exceeding GPU memory?

Accepted Answer

Enable gradient accumulation, processing 8 micro-batches of size 32 before performing a single optimizer step

Answer

Reduce the model's learning rate by a factor of 8 to compensate for the smaller batch size

Answer

Increase the number of training epochs by 8 times so the model sees the equivalent amount of data

Answer

Enable early stopping so training halts before GPU memory is fully consumed