Question

A GenAI engineer deploys a custom PyFunc RAG chain to a Databricks Model Serving endpoint. During a product launch, the endpoint receives sustained traffic of roughly 200 concurrent requests, but users report frequent HTTP 429 (Too Many Requests) errors even though CPU utilization on the compute is low. The endpoint is configured with a fixed compute size and a maximum concurrency well below the incoming load. What is the most appropriate action to eliminate the 429 errors while handling the sustained traffic?

Accepted Answer

Increase the endpoint's concurrency by scaling up the provisioned compute (larger workload size and/or higher max provisioned concurrency) so it can serve the sustained request volume.

Answer

Enable scale-to-zero on the endpoint so idle capacity is reclaimed and reallocated to active requests.

Answer

Switch the endpoint from CPU to GPU compute so each request completes faster and reduces queueing.

Answer

Reduce the retriever's top_k parameter in the chain to lower the payload size returned to clients.