Question

A data engineer is building a BigQuery ML logistic regression model to predict customer churn. The raw table contains a transaction timestamp, several engineered numeric features, and the churn label. During evaluation, the model reports 98% accuracy on the test set, but performance drops sharply in production. On review, the engineer notices they computed the mean and standard deviation for feature normalization across the entire table before running CREATE MODEL, and BigQuery ML randomly split rows into training and evaluation sets. What is the most likely cause of the inflated evaluation metrics, and how should it be fixed?

Accepted Answer

The normalization statistics were computed over the full dataset including evaluation rows, causing data leakage; use the TRANSFORM clause so preprocessing statistics are learned only from training data and applied consistently at prediction.

Answer

Random splitting is invalid for churn models; switch to DATA_SPLIT_METHOD='SEQ' on the timestamp and keep the manual normalization outside the model.

Answer

Logistic regression overfits by default; add L2 regularization with the l2_reg option and retain the precomputed normalization values.

Answer

The evaluation set is too small; increase DATA_SPLIT_EVAL_FRACTION to 0.5 and rerun the training with the same preprocessing pipeline.