Question

A data science team trains a gradient boosting model to predict loan defaults. During evaluation, they notice the model achieves 0.94 AUC on the training set but only 0.71 AUC on the validation set, and the gap has persisted across multiple runs. They want a principled way to quantify whether their model's true generalization performance is being reliably estimated before they commit to reducing this gap. Which action provides the MOST reliable estimate of generalization performance for this situation?

Accepted Answer

Use repeated k-fold cross-validation and report the mean and standard deviation of validation AUC across folds

Answer

Increase the number of boosting rounds so the training AUC rises further, then compare to validation AUC

Answer

Re-split the data with a different random seed and report whichever validation AUC is highest

Answer

Evaluate the model only on the training set since it contains the most data points