Question

A SaaS company runs a customer-support summarization feature on Amazon Bedrock using a large foundation model. Each request includes a 4,000-token static system prompt plus company knowledge base excerpts that are identical across all requests within a tenant, followed by a short unique conversation transcript. The team observes that time-to-first-token is dominated by processing the large repeated prefix, and per-request input token costs are high. Traffic is steady but the model choice must remain the same for quality reasons. Which optimization most directly reduces both latency and cost for this repeated-prefix workload with the least engineering effort?

Accepted Answer

Enable prompt caching on Bedrock so the static system prompt and repeated knowledge base prefix are cached and reused across requests

Answer

Purchase Provisioned Throughput model units to guarantee capacity for the large model

Answer

Use Bedrock model distillation to train a smaller student model that mimics the large model's summaries

Answer

Switch the requests to Bedrock batch inference to lower the per-token price of the repeated prefix