Question

A machine learning engineer is training a large deep learning model that requires 18 hours on a single ml.p3.8xlarge instance. Midway through a run, the training job failed due to a transient hardware fault, and the engineer lost all progress and had to restart from scratch. The team wants to minimize wasted compute if failures recur, without switching frameworks. Which approach best addresses this requirement?

Accepted Answer

Configure the training job to periodically save model checkpoints to Amazon S3 and load the latest checkpoint on job restart to resume training.

Answer

Split the dataset into smaller shards and run separate independent training jobs for each shard, then average the resulting model weights.

Answer

Enable SageMaker Debugger with a built-in rule to automatically resume the failed job from the point of interruption.

Answer

Increase the instance size to ml.p3.16xlarge so the training completes faster and reduces the probability of encountering a fault.