Question

A data engineer needs to score incoming IoT sensor readings in near-real-time. Records arrive continuously in a Delta table, and the team wants each new micro-batch to be scored by a model registered in Unity Catalog, with predictions written continuously to an output Delta table. Which approach correctly implements this streaming inference pipeline?

Accepted Answer

Load the model with mlflow.pyfunc.spark_udf(), read the source table with spark.readStream, apply the UDF to add a prediction column, and write results using writeStream with a Delta sink.

Answer

Deploy the model to a Databricks Model Serving endpoint and configure the endpoint to automatically consume the Delta table as a streaming source.

Answer

Load the model with mlflow.pyfunc.load_model() and call model.predict() inside a for-loop that polls the source table every few seconds.

Answer

Read the source table with spark.read (batch), apply the spark_udf, and schedule the notebook to run every minute as a Databricks Job.