Question

A data scientist is tuning a Spark ML pipeline using CrossValidator with a 3-fold cross-validation and a parameter grid of 4 combinations. After fitting, they want to inspect the individual model trained for every fold and every parameter combination (not just the final best model refit on all data) to analyze fold-level variance. By default, what does the fitted CrossValidatorModel expose, and what must be changed to access every fold's model?

Accepted Answer

By default only the best model (refit on the full dataset) and average metrics are retained; setting collectSubModels=True before fitting retains all per-fold sub-models in subModels.

Answer

By default all sub-models are always retained in the bestModel attribute; no configuration change is needed.

Answer

By default only the average metrics are retained; setting parallelism greater than 1 causes all per-fold models to be stored.

Answer

By default the CrossValidatorModel stores only the last fold's model; enabling checkpointing retains the rest.