Question

A media company runs a real-time content moderation service on Amazon Bedrock using Anthropic Claude. During peak evening hours, US traffic triples and the team observes rising ThrottlingExceptions even though their aggregate token volume across all regions is well below what a single region's on-demand quota could theoretically absorb if spread out. Their current setup invokes the model directly using a foundation model ID pinned to us-east-1. The team wants to increase effective throughput and reduce throttling WITHOUT requesting quota increases or committing to Provisioned Throughput. What is the MOST appropriate change?

Accepted Answer

Switch invocations to use a cross-region (system-defined) inference profile so requests are automatically distributed across multiple regions, pooling their on-demand capacity to absorb bursts.

Answer

Reduce the max_tokens parameter on every request so each call consumes fewer tokens and stays under the per-request throttling limit.

Answer

Enable prompt caching on the moderation prompt so repeated system context is not recharged against the throughput quota.

Answer

Move the workload to Bedrock batch inference to bypass on-demand throttling limits during peak hours.