Question

A company runs a customer-facing FAQ assistant built on a foundation model. Analytics show that roughly 40% of user questions are near-identical phrasings of a small set of common questions, and each identical query currently triggers a full model inference. Leadership wants to reduce cost and improve response latency for these repeated questions without degrading answer quality. Which approach best addresses this goal?

Accepted Answer

Implement a semantic response cache that stores answers for common questions and returns cached results for semantically similar incoming queries

Answer

Increase the temperature parameter so the model generates more varied answers for repeated questions

Answer

Fine-tune the foundation model on the full history of customer chat logs to memorize past answers

Answer

Switch the endpoint to a larger, more capable foundation model to answer common questions faster