Question

A data science team is training a large deep learning model that takes several days to complete on a multi-GPU cluster. Their infrastructure occasionally experiences node failures and scheduled maintenance windows. To ensure that a hardware interruption does not force them to restart training from the very beginning, which practice should they adopt?

Accepted Answer

Periodically save model checkpoints (weights and optimizer state) to persistent storage during training

Answer

Reduce the batch size so each training step completes faster and less work is lost per failure

Answer

Switch the workload from training to inference to avoid long-running jobs

Answer

Increase GPU memory allocation so the entire dataset stays resident in VRAM