Question

A financial services company serves a large transformer-based text classification model on Vertex AI endpoints backed by NVIDIA T4 GPUs. Traffic has tripled, and p99 latency now regularly exceeds the 150 ms SLA. The team wants to increase throughput per GPU without adding more replicas or retraining the model, and they can tolerate a very small drop in accuracy. Which optimization should they apply first?

Accepted Answer

Apply post-training INT8 quantization to the model and validate accuracy on a holdout set before redeploying.

Answer

Convert the model to FP64 precision to improve numerical stability during inference.

Answer

Enable knowledge distillation at serving time so the endpoint compresses requests dynamically.

Answer

Increase the endpoint's minReplicaCount so more GPUs share the load.