Question

You deploy a TensorFlow model on a Vertex AI endpoint using a custom serving container that runs a single inference server process per replica. Each replica is provisioned with 8 vCPUs and no GPU. Under load tests, individual requests complete in about 40 ms, but overall endpoint throughput plateaus far below what the CPU capacity suggests, and CPU utilization stays around 25%. Latency remains flat until the plateau, then rises sharply. You want to raise throughput per replica before adding more replicas. What is the most effective first step?

Accepted Answer

Increase the number of worker threads/inter-op parallelism in the serving container so a single replica processes multiple concurrent requests, then re-run the load test.

Answer

Immediately raise the endpoint's maximum replica count so autoscaling adds capacity to handle the plateau.

Answer

Switch the replicas to GPU accelerators to reduce the 40 ms per-request latency.

Answer

Reduce the model to int8 post-training quantization to cut per-request compute cost.