Question

A retail company is building a customer-facing chatbot that must respond to shoppers within about one second per turn. During prototyping, the team finds that their largest, most capable foundation model produces the highest-quality answers but consistently takes 4-6 seconds to respond. Product managers say the answer quality from a mid-sized model is acceptable for the simple product questions users ask most often. Which design change best addresses the requirement while controlling for user experience?

Accepted Answer

Switch the chatbot to the smaller, faster foundation model that meets the latency target with acceptable quality for the common questions.

Answer

Keep the largest model but increase the max token output limit so responses are complete.

Answer

Keep the largest model and raise the temperature parameter to speed up generation.

Answer

Keep the largest model but add more few-shot examples in the prompt to improve accuracy.