Question

A data scientist has a Spark DataFrame containing sales data for 500 different stores, identified by a store_id column. They want to train a separate lightweight scikit-learn regression model for each store in parallel across the cluster, returning a single DataFrame that summarizes each store's model coefficients. Which approach is the most appropriate?

Accepted Answer

Group the DataFrame by store_id and use applyInPandas, passing a function that receives each group as a pandas DataFrame, trains a scikit-learn model, and returns a pandas DataFrame of results

Answer

Use a scalar pandas UDF that trains one scikit-learn model per row batch and returns a Series of predictions

Answer

Collect the entire Spark DataFrame to the driver with .toPandas() and loop over each store_id in a Python for loop to train each model sequentially

Answer

Convert the DataFrame using the pandas API on Spark and call .groupby().apply() with a scikit-learn model fit inside the function