Question

A data engineer needs to score a 200-million-row Spark DataFrame using a registered scikit-learn model in Unity Catalog. Before prediction, each row must be enriched by calling a preprocessing function that depends on grouping records by 'store_id' to compute a rolling group-level statistic that the model expects as a feature. The engineer wants the enrichment and prediction to run in a single distributed pass. Which approach best satisfies these requirements?

Accepted Answer

Group the DataFrame by 'store_id' and use applyInPandas with a function that computes the group statistic and calls the loaded model's predict method

Answer

Load the model with mlflow.pyfunc.spark_udf and apply it directly to the DataFrame with a withColumn call

Answer

Collect the DataFrame to the driver, compute the group statistics in pandas, then broadcast the result back for scoring

Answer

Deploy the model to a Databricks Model Serving endpoint and send each row as a REST request from a Spark map operation