Question

A retail company runs a customer-facing chatbot on a large foundation model that produces high-quality responses but has high per-request cost and slow response times during peak traffic. The team wants to reduce inference cost and latency while preserving as much response quality as possible for their specific chatbot use case. Which approach best meets these goals?

Accepted Answer

Use model distillation to train a smaller student model using the large model's outputs, then deploy the smaller model for inference

Answer

Increase the temperature parameter on the large model to speed up response generation

Answer

Add more few-shot examples to every prompt so the large model returns answers faster

Answer

Enable provisioned throughput on the large model to permanently lower its per-token price