Question

You deployed a computer vision model to a Vertex AI online prediction endpoint backed by GPU-equipped nodes. Traffic is bursty: during business hours request volume spikes 10x, but at night it drops to near zero. During spikes, p99 latency exceeds your 200 ms SLO, while at night you are paying for idle GPU nodes. You want to keep latency within SLO during spikes while minimizing cost during quiet periods, with minimal ongoing management. What should you do?

Accepted Answer

Configure autoscaling on the DeployedModel with a GPU-duty-cycle target metric, and set minReplicaCount to a low value with a higher maxReplicaCount.

Answer

Set both minReplicaCount and maxReplicaCount to the peak number of replicas needed so capacity is always available for spikes.

Answer

Switch the endpoint to batch prediction jobs scheduled hourly so you only pay for compute while jobs run.

Answer

Deploy the model to a single large GPU machine type and rely on manual redeployment to add replicas when latency alerts fire.