Question

A retail company runs a 340M-parameter BERT-based text classifier on a Vertex AI endpoint to categorize incoming customer chat messages in real time. The model meets accuracy targets, but p99 latency is 480 ms — well above the 120 ms SLA — and GPU serving costs are high. The team wants to dramatically reduce latency and cost while keeping accuracy within about 2% of the current model, and they are willing to invest in a one-time offline effort. Which approach best meets these requirements?

Accepted Answer

Apply knowledge distillation to train a much smaller student model using the current model as the teacher, then serve the student model.

Answer

Switch the endpoint from online inference to batch inference to process messages more efficiently.

Answer

Increase the minimum replica count and add more GPUs to the endpoint's autoscaling configuration.

Answer

Enable request-response logging and raise the endpoint's maximum concurrent request limit.