Question

A data scientist has a training dataset of about 500 MB that comfortably fits in the memory of a single Databricks driver node. They want to train a gradient boosted tree model and need the fastest iteration time with the least overhead. Which training approach is most appropriate for this situation?

Accepted Answer

Use a single-node library like scikit-learn on the driver, since distributed training adds coordination overhead that outweighs its benefits for small data

Answer

Use Spark ML's GBTClassifier so the training is automatically parallelized across all worker nodes for maximum speed

Answer

Convert the pandas DataFrame to a Spark DataFrame and use applyInPandas to distribute the training across partitions

Answer

Use a pandas UDF to train independent models on each Spark partition and average their predictions