Question

A data science team is comparing three different preprocessing strategies (raw features, PCA-reduced features, and target-encoded features) for a churn prediction model. For each strategy they run several XGBoost training jobs with different hyperparameters. They need a single, organized way to group all runs by preprocessing strategy, compare validation AUC across every run, and later reproduce the exact configuration that produced the best result. Which approach best meets these requirements with the least custom engineering?

Accepted Answer

Create one SageMaker Experiment with a separate Run for each preprocessing strategy, logging hyperparameters and validation AUC as run metrics, then compare runs in SageMaker Studio's experiment analytics view.

Answer

Store each training job's hyperparameters and AUC as tags on the training job, then use a CloudWatch dashboard to visualize and compare the tagged values.

Answer

Write the hyperparameters and validation AUC of every job to a DynamoDB table and build a custom Athena query to identify the best configuration.

Answer

Register every trained model as a new version in the SageMaker Model Registry and rely on the approval status field to indicate which preprocessing strategy performed best.