Question

You run a TensorFlow image-classification model on a Vertex AI online endpoint backed by GPUs. Traffic is bursty, and during peak periods GPU utilization stays low (around 30%) even though request latency climbs and clients report timeouts. Each individual request contains a single image. You want to increase throughput and improve GPU efficiency without reducing model accuracy or provisioning more replicas. What is the most effective change?

Accepted Answer

Enable server-side request (dynamic) batching so multiple concurrent single-image requests are grouped into one batch before the GPU forward pass.

Answer

Switch the endpoint from GPU to CPU machines to eliminate the GPU scheduling overhead causing the low utilization.

Answer

Convert the deployment to a batch prediction job that reads requests from Cloud Storage and writes results back.

Answer

Increase the autoscaling maximum replica count and lower the target GPU utilization threshold to 20%.