Question

A data scientist is iterating on an XGBoost churn model in SageMaker. Over two weeks they launch dozens of training jobs with different feature sets, hyperparameters, and preprocessing steps. Their manager asks them to produce a side-by-side comparison of validation AUC across all runs, grouped by which feature set was used, so the team can decide which configuration to promote. The scientist wants a reproducible, low-effort way to track and later analyze these runs without manually recording results in a spreadsheet. Which approach best meets this requirement?

Accepted Answer

Use SageMaker Experiments to log each training job as a run within an experiment, recording parameters and metrics, then compare runs in the analytics/comparison view.

Answer

Enable SageMaker Debugger on every training job and inspect the tensor collections to reconstruct validation AUC for each configuration.

Answer

Register each trained model as a version in the SageMaker Model Registry and compare their metadata in the registry console.

Answer

Write a custom Lambda function that parses each training job's CloudWatch Logs to extract AUC values and stores them in a DynamoDB table.