Question

A data scientist has a training dataset of 2 GB that fits comfortably in the memory of a single Databricks driver node. They want to train a scikit-learn RandomForestClassifier while still distributing the hyperparameter search across the workers of the cluster. Which approach best accomplishes this goal?

Accepted Answer

Keep the data as a pandas DataFrame, train with scikit-learn, and use Hyperopt with SparkTrials to distribute the hyperparameter search across workers

Answer

Convert the data to a Spark DataFrame and train with Spark ML's RandomForestClassifier inside a Pipeline

Answer

Keep the data as a pandas DataFrame and use scikit-learn's GridSearchCV with n_jobs=-1 on the driver only

Answer

Use applyInPandas to broadcast the entire dataset to every worker and train an independent scikit-learn model per partition