Question

A data scientist has a 2 GB training dataset that fits comfortably in the memory of a single Databricks worker. They want to train a gradient-boosted tree model using scikit-learn and are unsure whether to convert their Spark DataFrame to a pandas DataFrame or keep the data distributed. Which statement correctly describes the most appropriate approach for this situation?

Accepted Answer

Convert the Spark DataFrame to a pandas DataFrame with toPandas() and train the scikit-learn model on the driver, since single-node training is appropriate for data that fits in memory.

Answer

Keep the data as a Spark DataFrame and train the scikit-learn model directly, because scikit-learn estimators natively operate on distributed Spark DataFrames.

Answer

Use the pandas API on Spark and pass the resulting object directly to scikit-learn's fit() method to achieve distributed training of the scikit-learn model.

Answer

Repartition the Spark DataFrame across all workers and call fit() so scikit-learn automatically parallelizes the gradient-boosted tree training across the cluster.