Question

You are building a multi-turn customer support agent on Databricks that uses a foundation model with an 8,192-token context window. Each turn appends the full conversation history plus a system prompt plus retrieved documents. After roughly 15 exchanges, users start receiving errors that the context length has been exceeded. The business requires that the agent still remember key facts from earlier in the conversation (like the customer's account ID and reported issue). What is the most appropriate way to manage context so the agent keeps working across long conversations?

Accepted Answer

Use a conversation summary memory that progressively condenses older turns into a running summary while keeping the most recent turns verbatim.

Answer

Increase the max_tokens generation parameter so the model has more room to produce output.

Answer

Truncate the conversation by dropping the oldest turns entirely once the context limit is approached.

Answer

Switch to a larger provisioned-throughput endpoint of the same model to raise the throughput limit.