Question

A data scientist has a training dataset of 800 MB that comfortably fits into the memory of a single Databricks worker node. They want to train a scikit-learn RandomForestClassifier and perform a large hyperparameter search over 500 candidate configurations. They want to distribute the individual training trials across the cluster's worker nodes to speed up the search. Which approach best meets this requirement?

Accepted Answer

Use Hyperopt with SparkTrials to distribute the hyperparameter trials across the cluster while training each model with scikit-learn on a single node

Answer

Convert the data to a Spark DataFrame and use Spark ML's CrossValidator, since scikit-learn cannot run on a Databricks cluster

Answer

Use scikit-learn's GridSearchCV with n_jobs=-1, which automatically distributes trials across all worker nodes in the cluster

Answer

Use Hyperopt with the default Trials class, which distributes each trial to a separate Spark executor