Question

A data engineer needs to score a 500-million-row Spark DataFrame using a scikit-learn model that was logged with MLflow and registered in Unity Catalog as `main.ml.churn_model` with the alias `champion`. The scoring must run as a distributed batch job on a Spark cluster, and the model expects several feature columns already present in the DataFrame. Which approach correctly applies the registered model for distributed batch inference?

Accepted Answer

Load the model as a Spark UDF with `mlflow.pyfunc.spark_udf(spark, 'models:/main.ml.churn_model@champion')` and apply it to the feature columns with `df.withColumn('prediction', udf(*feature_cols))`.

Answer

Load the model with `mlflow.sklearn.load_model('models:/main.ml.churn_model@champion')` and call `model.predict(df)` directly on the Spark DataFrame.

Answer

Deploy the model to a Databricks Model Serving endpoint and issue one REST request per row of the DataFrame from the driver.

Answer

Convert the Spark DataFrame to a pandas DataFrame with `df.toPandas()`, then call `model.predict()` on the resulting pandas object.