Question

A data scientist builds a Spark ML Pipeline with three stages: a StringIndexer, a StandardScaler, and a LogisticRegression estimator. They want to tune the regularization parameter using k-fold cross-validation while ensuring the StandardScaler is refit on only the training folds during each CV iteration. What is the correct way to configure this?

Accepted Answer

Pass the entire Pipeline object as the estimator to CrossValidator, and build the ParamGrid referencing the LogisticRegression stage's regParam.

Answer

Fit the Pipeline on the full dataset first, then pass only the fitted LogisticRegression stage to CrossValidator.

Answer

Pass only the LogisticRegression estimator to CrossValidator and apply the StringIndexer and StandardScaler to the full dataset beforehand.

Answer

Wrap CrossValidator inside the Pipeline as its final stage so tuning happens automatically during Pipeline.fit().