Question

A financial services company runs a customer support assistant on Amazon Bedrock using Anthropic Claude via the Converse API. Every request includes a large, static 8,000-token system prompt containing compliance policies, tone guidelines, and 15 few-shot examples, followed by a short user question (typically under 100 tokens). Traffic is high and consistent throughout the business day. The team notices that input token costs dominate their bill and that per-request latency is higher than expected. They want to reduce both input token cost and time-to-first-token without removing the compliance context or degrading answer quality. Which approach best addresses this?

Accepted Answer

Enable Bedrock prompt caching by marking the static system prompt and few-shot block with a cache checkpoint so repeated invocations reuse the cached prefix.

Answer

Move the entire system prompt into a Bedrock Knowledge Base and retrieve it per request so the tokens are billed at the lower retrieval rate.

Answer

Switch from the Converse API to batch inference so all requests share a single amortized token cost.

Answer

Fine-tune the model on the compliance policies and few-shot examples so the static context no longer needs to be sent in each request.