Question

Your team trains a large PyTorch image model on Vertex AI custom training. Each run takes about 30 hours on a single node with multiple GPUs. To reduce cost, you switched to Spot (preemptible) VMs, but you notice that when a VM is reclaimed, the job restarts from scratch and wastes many GPU-hours. You must keep costs low while ensuring interrupted runs can resume efficiently. What should you do?

Accepted Answer

Periodically save model and optimizer state to Cloud Storage and configure the training job to resume from the latest checkpoint on restart, keeping Spot VMs and enabling automatic retries.

Answer

Switch back to standard (on-demand) VMs and remove checkpointing since the job will no longer be interrupted.

Answer

Reduce the number of training epochs so that each run completes before a Spot VM is likely to be preempted.

Answer

Move the training to a smaller CPU-only machine so that preemption happens less frequently and restarts are cheaper.