Question

Your team serves a large image-classification model on a Vertex AI online endpoint backed by GPUs. Profiling shows the GPUs sit at only 30% utilization while p95 latency remains high. Investigation reveals that each request spends most of its time in a heavy Python-based image decode and resize step that runs on the same GPU containers before inference. The GPU compute for the actual forward pass is fast. You must increase throughput and reduce latency without buying more GPUs. What is the most effective approach?

Accepted Answer

Move the image decode and resize preprocessing into a separate CPU-based service (or CPU node pool) that feeds the GPU inference service, so GPUs only handle the forward pass.

Answer

Increase the endpoint's max replica count so more GPU containers spin up to absorb the load.

Answer

Apply INT8 post-training quantization to the model to make the forward pass faster.

Answer

Switch the endpoint from REST to gRPC to reduce per-request serialization overhead.