Question

A data scientist is tuning a Spark ML classification pipeline using CrossValidator with a 4-parameter ParamGridBuilder (36 total combinations) and 3 folds. The job runs slowly because model fits are executing sequentially, even though the cluster has many idle worker cores. Which single change will let CrossValidator evaluate multiple model fits concurrently to better utilize the cluster?

Accepted Answer

Set the CrossValidator's parallelism parameter to a value greater than 1

Answer

Increase numFolds so more folds are evaluated at once

Answer

Replace CrossValidator with TrainValidationSplit to enable parallel fits

Answer

Repartition the training DataFrame into more partitions before fitting