Question

A retail company deploys a generative AI customer-service assistant built on a large language model billed per token. During a quarterly review, finance flags that costs are far higher than projected, even though the number of customer conversations matches the forecast. The AI leader reviews the implementation and finds that each customer query prepends a 3,000-word product catalog and full conversation history into every prompt. Which change would most directly reduce the token-based cost while preserving answer quality?

Accepted Answer

Use retrieval-augmented generation to inject only the most relevant catalog snippets and summarize older conversation turns

Answer

Switch to a fine-tuned model so the assistant no longer requires any prompt input

Answer

Lower the model's temperature setting to make responses shorter and more deterministic

Answer

Increase the maximum output token limit so responses complete in a single request