Question

A media company runs a customer-support summarization workload on Amazon Bedrock using Anthropic Claude 3.5 Sonnet. The summaries are high quality, but at 4 million requests per month the inference cost is too high and per-request latency exceeds their 800ms SLA. They have a large corpus of historical support tickets with the Sonnet-generated summaries already stored. They want to reduce both cost and latency while keeping summary quality as close as possible to the current output, with minimal ongoing ML engineering effort. Which approach best meets these requirements?

Accepted Answer

Use Amazon Bedrock Model Distillation to fine-tune a smaller Claude Haiku student model using the Sonnet teacher outputs, then serve the distilled model.

Answer

Enable prompt caching on the Sonnet model and add a CloudFront distribution in front of the Bedrock endpoint to cache responses.

Answer

Switch to a Retrieval Augmented Generation pipeline with a Bedrock Knowledge Base so the model retrieves prior summaries instead of generating new ones.

Answer

Purchase Provisioned Throughput for the Sonnet model to guarantee capacity and lower the effective per-token price.