Question

Your team serves a large vision transformer model on a Vertex AI online endpoint. The model receives sustained, high-volume traffic during business hours, and each request runs a compute-intensive forward pass. Current single-GPU replicas hit their throughput ceiling and requests queue, driving up latency, even though CPU and memory usage remain low. You need to increase per-replica throughput while keeping per-inference cost reasonable. What is the most effective first step?

Accepted Answer

Attach a more powerful accelerator (e.g., move from an older GPU to an NVIDIA A100 or L4) to the endpoint's machine type so each replica processes the compute-bound forward pass faster.

Answer

Increase the machine type's vCPU and memory allocation so the model has more resources for the forward pass.

Answer

Switch the endpoint from online prediction to batch prediction to avoid request queuing.

Answer

Raise the endpoint's maximum replica count and lower the target GPU utilization so autoscaling adds replicas sooner.