Question

A data engineering team must score 40 million customer records once every night using a model registered in Unity Catalog. The scoring output feeds a downstream reporting table, and there is no requirement for sub-second per-record latency. The team wants the most efficient and cost-appropriate way to run this workload on Databricks. Which approach should they choose?

Accepted Answer

Load the model as a Spark UDF with mlflow.pyfunc.spark_udf and apply it to the Spark DataFrame in a scheduled batch job

Answer

Deploy the model to a Databricks Model Serving endpoint and send each record via REST calls in a loop

Answer

Configure a Structured Streaming query with a foreachBatch sink that queries a serving endpoint for each micro-batch

Answer

Register the model as a real-time endpoint with scale-to-zero enabled and trigger it from a nightly job