Question

A data scientist is tuning hyperparameters for a scikit-learn gradient boosting model on a single large Databricks cluster. The training dataset fits comfortably in the driver's memory, and each individual model trains quickly on one machine. They want to use Hyperopt to evaluate many hyperparameter combinations and distribute those evaluations across the cluster's worker nodes to finish the search faster. Which Hyperopt configuration should they use in the fmin() call?

Accepted Answer

Pass trials=SparkTrials() so that individual hyperparameter evaluations are distributed across the cluster's workers.

Answer

Pass trials=Trials() because it automatically parallelizes evaluations across Spark executors.

Answer

Convert the pandas DataFrame to a Spark DataFrame and train a Spark ML model so Hyperopt can distribute the data.

Answer

Wrap the training function in a pandas UDF and pass trials=Trials() to run each trial on a separate partition.