Question

Your team serves a large BERT-based text classifier on a Vertex AI endpoint with GPU accelerators. Traffic has grown, and GPU serving costs are becoming unsustainable. The business requires that classification accuracy remain within 1% of the current model, and that p95 latency stay under 50 ms. You want to reduce serving cost the most while meeting both constraints, and you have access to the full labeled training dataset plus abundant unlabeled production text. Which optimization approach should you pursue first?

Accepted Answer

Train a much smaller student model via knowledge distillation using the large model as the teacher, then evaluate it against the accuracy and latency constraints on CPU serving.

Answer

Apply post-training int8 quantization to the existing large model and deploy it unchanged on the same GPU endpoint.

Answer

Increase the endpoint's minimum replica count and lower the autoscaling GPU utilization target to spread traffic across more nodes.

Answer

Switch the endpoint from GPU to a larger-memory GPU type to process more requests per replica.