Question

A data engineering team ingests transaction events into a Delta table that is continuously updated. They want to score each incoming micro-batch of records in near real time using a model that was logged with MLflow and registered in Unity Catalog. They plan to read the Delta table as a streaming DataFrame and apply the model. Which approach correctly enables continuous scoring of the streaming data with the registered model?

Accepted Answer

Load the model as a Spark UDF with mlflow.pyfunc.spark_udf() and apply it to the streaming DataFrame read via spark.readStream, then write results out with writeStream.

Answer

Deploy the model to a Databricks Model Serving endpoint and query it once per hour with a scheduled batch job.

Answer

Load the model with mlflow.pyfunc.load_model() and call model.predict() directly on the streaming DataFrame, then use df.write.save() to persist results.

Answer

Convert the streaming DataFrame to a pandas DataFrame with toPandas() and score it in a loop inside a foreachBatch handler.