Question

A data science team trains two versions of a binary classifier on the same small validation set of 400 records. Version A reports 88.0% accuracy and Version B reports 88.5% accuracy across a single evaluation run. The lead engineer must decide whether Version B is genuinely better before promoting it in the SageMaker Model Registry. What is the most statistically sound next step?

Accepted Answer

Use k-fold cross-validation to obtain a distribution of scores for each version and compare their mean and variance rather than relying on a single point estimate.

Answer

Promote Version B immediately because its accuracy is numerically higher on the validation set.

Answer

Retrain both versions with a larger learning rate to amplify the accuracy difference and make the comparison clearer.

Answer

Switch the evaluation metric from accuracy to training loss and compare the final loss values of both versions.