Question

A data science team at a lending company is building a churn model in Vertex AI. Their training labels come from an event table where each customer has a churn event with a specific timestamp. Their features (account balance, transaction counts, support tickets) are stored in a Vertex AI Feature Store and change frequently over time. When they built their first training set by joining the latest feature values to each label, the model performed well in offline evaluation but poorly in production. What is the most likely cause and the correct fix?

Accepted Answer

The training set was built with feature values from after the label timestamp, causing data leakage; use a point-in-time lookup (fetch_historical_feature_values with the label event timestamps) so each label sees only feature values as of its event time.

Answer

The features were not scaled; apply standardization to all numeric features before joining them to the labels.

Answer

The Feature Store online store was used for training; switch to reading from the online store to guarantee freshness during training.

Answer

The training set was too small; increase the number of labels and retrain without changing the join logic.