Question

A data science team runs a distributed training job on a SageMaker training cluster of four ml.p3.8xlarge GPU instances. The job completes successfully but takes far longer than expected, and costs are higher than budgeted. To identify whether the expensive GPU resources are being fully used before the next training run, which approach requires the LEAST setup effort and provides the most direct visibility?

Accepted Answer

Enable SageMaker Debugger profiling and review the GPUUtilization and GPUMemoryUtilization metrics that SageMaker publishes to CloudWatch under the /aws/sagemaker/TrainingJobs namespace

Answer

Install a custom nvidia-smi polling script inside the training container and stream its output to a self-managed Prometheus and Grafana stack on EC2

Answer

Enable SageMaker Model Monitor data quality baselining to detect utilization drift across training runs

Answer

Turn on VPC Flow Logs for the training subnet and analyze inter-node traffic in CloudWatch Logs Insights