Question

A GenAI engineer has deployed a customer-facing RAG chatbot on a Databricks Model Serving endpoint. During a promotional campaign, a small number of automated clients begin flooding the endpoint with thousands of requests, causing legitimate users to experience high latency and occasional 429 errors. The team wants to protect overall availability by ensuring no single client can consume a disproportionate share of endpoint capacity, without rearchitecting the application. What should the engineer configure?

Accepted Answer

Set per-endpoint and per-user rate limits on the Model Serving endpoint to cap the number of queries per minute an individual identity can send.

Answer

Enable scale-to-zero on the endpoint so idle capacity is released and reclaimed when the abusive clients stop sending requests.

Answer

Increase the provisioned concurrency to the maximum allowed value so the endpoint can absorb all incoming traffic regardless of source.

Answer

Add an A/B traffic split routing 50% of requests to a second model version to distribute the load across two models.