Question

A data scientist is preparing a churn prediction dataset with AWS Glue. One categorical feature, 'sales_region', has a strong relationship with the target label, and the team wants to replace each region with the mean churn rate for that region (target encoding). During initial experiments, cross-validation accuracy is excellent but the model performs poorly on held-out production data. What is the MOST likely cause and the correct way to fix it?

Accepted Answer

The target encoding was computed over the entire dataset including validation rows, causing target leakage; compute the encoding only from the training fold using out-of-fold or smoothed statistics.

Answer

The feature has too many categories; apply one-hot encoding instead to eliminate the discrepancy between validation and production.

Answer

The numeric encoded values were not standardized; apply min-max scaling to the encoded column before training.

Answer

The Glue crawler inferred the wrong data type for 'sales_region'; recast the column to string and re-run the encoding.