Question

Your team runs a fraud-scoring model on a Vertex AI online endpoint. Traffic is steady during business hours but drops to near-zero overnight. During morning traffic spikes, users report occasional latency violations of your 100 ms p99 SLO, which correlate with autoscaling events that add new replicas. Cost is a secondary concern; meeting the SLO consistently is the priority. Which configuration change best addresses the latency violations?

Accepted Answer

Increase the minimum replica count so enough warm replicas are always provisioned to absorb the morning spike without waiting for scale-up

Answer

Lower the CPU utilization autoscaling target so replicas are added earlier, and keep the minimum replica count at zero to save overnight cost

Answer

Switch the endpoint from online inference to a scheduled batch prediction job that runs each morning before traffic arrives

Answer

Enable dynamic request batching to combine incoming requests into larger batches for higher throughput