Question

A data engineer needs to score 500 million records in a nightly Spark batch job using an MLflow model registered in Unity Catalog. Each record must first be grouped by 'store_id' so that a store-specific normalization step (implemented as custom Python logic requiring the full group in memory) is applied before the model's predictions are generated. Which approach correctly handles the per-group preprocessing and inference in a distributed manner?

Accepted Answer

Group the DataFrame by 'store_id' and use applyInPandas with a function that performs the per-group normalization and then calls the loaded model to predict, returning a pandas DataFrame with a defined output schema.

Answer

Use mlflow.pyfunc.spark_udf to load the model and call it directly on the DataFrame, relying on the UDF to group and normalize the data automatically.

Answer

Collect the entire DataFrame to the driver with .toPandas(), apply normalization per group in a loop, and score with the model in memory.

Answer

Register the model as a spark_udf and wrap it in a Structured Streaming query so each store's records are processed as a separate micro-batch.