Question

A company runs a customer-facing chatbot on a foundation model accessed through a pay-per-token API. During a cost review, the finance team notices that monthly charges have grown sharply even though the number of conversations stayed flat. Engineers discover that each request now injects an entire 40-page policy manual into the prompt and asks the model to produce a long, detailed answer. Which change would MOST directly reduce cost per request?

Accepted Answer

Trim the injected context to only the relevant policy sections and cap the response length so fewer input and output tokens are processed

Answer

Increase the temperature parameter so the model generates more creative responses

Answer

Switch the endpoint from real-time inference to provisioned throughput without changing the prompts

Answer

Enable multimodal input so the manual can be uploaded as an image instead of text