Question

A GenAI engineer has two versions of a RAG chatbot registered in Unity Catalog: version 3 (currently in production) and version 4 (a newly tuned prompt template). Before fully promoting version 4, the team wants to route 90% of live traffic to version 3 and 10% to version 4 to compare real-world quality and latency. How should the engineer configure this on a single Databricks Model Serving endpoint?

Accepted Answer

Create two served entities on the same endpoint (one for each model version) and set traffic_percentage to 90 and 10 respectively in the endpoint's traffic config.

Answer

Deploy each model version to its own separate serving endpoint and use an external load balancer to split traffic 90/10.

Answer

Set an MLflow alias 'production' on version 3 and 'canary' on version 4, then the endpoint automatically sends 10% to the canary alias.

Answer

Log both versions into a single MLflow pyfunc model and use random sampling inside the model code to pick the version per request.