Question

A data engineer registered a scikit-learn model in Unity Catalog and wants to score a 200-million-row Delta table stored in the lakehouse. Latency is not a concern, but the job must finish within a scheduled nightly window and leverage the full cluster. Which approach best fits these requirements?

Accepted Answer

Load the model with mlflow.pyfunc.spark_udf() and apply it as a column expression across the Spark DataFrame

Answer

Deploy the model to a Databricks Model Serving endpoint and send each row as a REST request in a loop

Answer

Load the model with mlflow.pyfunc.load_model() on the driver and call .predict() on a pandas conversion of the full table

Answer

Read the table as a Structured Streaming source and apply the model with a stateful streaming aggregation