Question

A data scientist is tuning a Spark ML GBTRegressor on a very large training DataFrame (over 200 million rows). They build a ParamGridBuilder with 12 hyperparameter combinations. Model training on this dataset is computationally expensive, and the team wants to keep total tuning time reasonable while still getting a defensible estimate of generalization performance. Which tuning approach best fits these constraints?

Accepted Answer

Use TrainValidationSplit, which trains each of the 12 parameter combinations once against a single holdout validation split.

Answer

Use CrossValidator with numFolds=10 so that each combination is trained 10 times for a robust estimate.

Answer

Use CrossValidator with numFolds=5, accepting that each of the 12 combinations is trained 5 times.

Answer

Skip validation entirely and fit all 12 combinations on the full dataset, then pick the one with the lowest training RMSE.