Question

A data scientist builds a Spark ML Pipeline with a StringIndexer, a VectorAssembler, and a RandomForestClassifier. They wrap the entire Pipeline inside a CrossValidator with a 3-fold configuration and a ParamGrid containing 4 combinations of numTrees. After calling cv_model = crossValidator.fit(trainDF), they want to use the resulting model for scoring new data. Which object should they call .transform() on, and what does it represent?

Accepted Answer

cv_model.bestModel, which is the full PipelineModel refit on all of trainDF using the best-performing hyperparameter combination

Answer

cv_model directly, but it only contains the RandomForestClassifier stage and requires re-fitting the indexer separately

Answer

The last of the 12 sub-models trained during cross-validation, since it saw the most data

Answer

cv_model.avgMetrics, which holds the trained model averaged across the 3 folds