Question

You are training a large image classification CNN on a single Vertex AI custom training node equipped with one NVIDIA A100 (40 GB) GPU. Training crashes with a CUDA out-of-memory error only after you increased the input image resolution from 224x224 to 512x512. The model architecture and dataset are fixed, and you must keep the higher resolution. You want the simplest change that lets training proceed on the same single-GPU node without significantly degrading final model accuracy. Which action should you take first?

Accepted Answer

Reduce the per-step batch size and use gradient accumulation to maintain the effective batch size

Answer

Switch the training node to a machine with 8 A100 GPUs and enable MirroredStrategy

Answer

Downsample the input images back to 224x224 to fit within GPU memory

Answer

Convert the model to run on a TPU v3 Pod slice to avoid GPU memory limits