Question

A data scientist is preparing a Spark DataFrame for a regression model. They plan to standardize a numeric feature using a StandardScaler (subtracting the mean and dividing by the standard deviation). To avoid data leakage while producing an honest estimate of generalization performance, when should the mean and standard deviation used for scaling be computed?

Accepted Answer

Fit the scaler on the training set only, then use those learned statistics to transform both the training and test sets

Answer

Fit the scaler on the entire dataset before splitting into training and test sets, then split

Answer

Fit a separate scaler independently on the training set and on the test set using each set's own statistics

Answer

Fit the scaler on the test set only, then apply those statistics to the training set