Question

A data scientist is training a gradient-boosted tree classifier on a 500 GB dataset stored in a Delta table that does not fit in the memory of a single worker. They want to tune 6 continuous and integer hyperparameters efficiently, exploring the search space adaptively rather than exhaustively, while keeping all training distributed across the cluster. Which approach best fits these requirements?

Accepted Answer

Use Hyperopt's fmin with the TPE algorithm, where the objective function fits a Spark ML pipeline on the full distributed dataset, running trials sequentially

Answer

Use Spark ML's estimator with CrossValidator and a ParamGridBuilder specifying all combinations of the 6 hyperparameters

Answer

Use Hyperopt's fmin with the TPE algorithm and SparkTrials, wrapping a scikit-learn model trained on a pandas DataFrame loaded from the Delta table

Answer

Use Spark ML's estimator with TrainValidationSplit and a ParamGridBuilder covering the full grid of the 6 hyperparameters