Question

A data scientist builds an XGBoost model to predict customer churn. During training, cross-validation reports 96% accuracy, but when the model is deployed and evaluated on genuinely new customers, accuracy drops to 71%. Investigation reveals that the training pipeline computed feature scaling statistics (mean and standard deviation) using the entire dataset before splitting into folds, and one input feature was derived from a value only known after churn occurs. What is the MOST likely root cause of the inflated cross-validation score?

Accepted Answer

Data leakage from fitting preprocessing on the full dataset and including a target-derived feature

Answer

The learning rate was set too high, causing the model to diverge during training

Answer

Insufficient regularization causing high variance that only affects new data

Answer

The cross-validation used too few folds, producing an optimistic estimate