Question

A data engineer needs to score 40 million records nightly using a scikit-learn model logged with MLflow and registered in Unity Catalog. Each prediction depends only on that row's feature values (no grouping or per-group state), and the input DataFrame is a Spark DataFrame. The team wants the simplest approach that distributes the scoring across the cluster without writing custom grouping logic. Which approach should they use?

Accepted Answer

Load the model as a Spark UDF with mlflow.pyfunc.spark_udf() and apply it to the feature columns in a withColumn call.

Answer

Use df.groupBy().applyInPandas() with a function that loads the model and returns predictions for each group.

Answer

Convert the Spark DataFrame to a pandas DataFrame with toPandas() and call model.predict() on the driver.

Answer

Deploy the model to a Databricks Model Serving endpoint and send each row as a REST request.