Question

Your team serves a fraud-scoring model on a Vertex AI online endpoint. Median (p50) latency is well within your 40 ms SLO, but the p99 latency regularly spikes to 300 ms, causing occasional timeouts for the payment service. CPU utilization is moderate and autoscaling is not triggering. Investigation shows the spikes correlate with garbage collection pauses and large individual request payloads that occasionally arrive together. Which action most directly reduces the p99 tail latency without over-provisioning?

Accepted Answer

Enforce a maximum request payload size and route oversized/batch requests to a separate batch prediction pipeline, isolating latency-sensitive online traffic

Answer

Increase the autoscaling target CPU utilization so replicas scale out earlier under load

Answer

Switch the endpoint from online prediction to Vertex AI batch prediction to eliminate per-request overhead

Answer

Reduce the number of min replicas to consolidate traffic onto fewer, larger machines