Question

A data scientist has a Spark DataFrame containing sales data for 500 retail stores, with columns store_id, week, features, and target. They want to train a separate scikit-learn regression model for each store in parallel across the cluster, returning a DataFrame with one row per store containing the store_id and its model's RMSE. Which approach correctly accomplishes this in a single distributed operation?

Accepted Answer

Group the DataFrame by store_id and call applyInPandas, passing a function that trains a scikit-learn model on the group's pandas DataFrame and returns a pandas DataFrame, along with an explicit output schema.

Answer

Use a pandas_udf of type SCALAR that accepts the features column and returns predictions, then aggregate the RMSE per store afterward.

Answer

Collect the Spark DataFrame to the driver with toPandas(), loop over each store_id, and train a scikit-learn model for each in a Python for-loop.

Answer

Convert the DataFrame using the pandas API on Spark and call .groupby('store_id').apply() with a scikit-learn training function.