Question

A data scientist runs a Spark ML CrossValidator with a 4-fold configuration over a ParamGridBuilder that contains 3 candidate values for regParam. After calling cv.fit(trainingDF), they want to programmatically identify which regParam value performed best across the folds. Which attribute of the fitted CrossValidatorModel should they inspect to make this decision?

Accepted Answer

avgMetrics, which holds the average cross-validation metric for each ParamMap in the grid

Answer

bestModel.stages, which lists the transformer stages of the winning pipeline

Answer

subModels, which stores every model trained on every fold

Answer

summary.objectiveHistory, which tracks the loss at each training iteration