Question

You deployed a large PyTorch recommendation model to a Vertex AI endpoint with autoscaling configured to scale down to zero replicas during off-peak hours to save cost. Users report that the first requests after idle periods experience 15+ second latencies, while steady-state requests return in under 200 ms. Your SLA requires p99 latency below 1 second at all times. What is the most effective change to meet the SLA?

Accepted Answer

Set the minimum replica count to at least 1 so a warm instance is always available to serve requests.

Answer

Increase the maximum replica count and lower the CPU utilization autoscaling target.

Answer

Enable dynamic request batching on the endpoint to group incoming requests.

Answer

Switch the endpoint from GPU to CPU machines to reduce container startup time.