Question

A data scientist runs a SageMaker automatic model tuning job for an XGBoost classifier. The tuning job is configured to maximize an objective metric computed on the training set. After the job finishes, the best trial shows 99% training accuracy, but when the chosen hyperparameters are used to train a final model and deploy it, real-world performance is poor. What is the most likely cause and the best corrective action?

Accepted Answer

The tuning job optimized a training-set metric, so it selected hyperparameters that overfit; reconfigure the objective metric to be computed on a held-out validation channel.

Answer

The instance type used for tuning was too small; switch to a larger GPU instance and rerun the tuning job.

Answer

The number of tuning jobs was too low; increase MaxNumberOfTrainingJobs to explore more of the hyperparameter space.

Answer

XGBoost cannot be used with automatic model tuning; switch to the SageMaker Linear Learner algorithm instead.