Question

A financial services company is preparing to launch a customer-facing chatbot built on an Amazon Bedrock custom fine-tuned model. They have decided to purchase Provisioned Throughput with a 1-month commitment to guarantee consistent latency during business hours. The launch is scheduled for Monday at 9:00 AM, and the operations team plans to submit the Provisioned Throughput purchase request programmatically at 8:55 AM Monday morning to align billing with the launch. During a rehearsal, the team observes that the model unit is not immediately available for inference after the create request returns. What should the team do to ensure the endpoint is ready to serve traffic at launch?

Accepted Answer

Submit the Provisioned Throughput purchase request in advance, because provisioning a model unit takes time to complete and the ProvisionedModelStatus must reach 'InService' before invocations succeed.

Answer

Add exponential backoff with jitter to the InvokeModel calls so requests automatically retry until the model unit finishes provisioning at 9:00 AM.

Answer

Switch to on-demand throughput for the first hour and cut over to Provisioned Throughput once the endpoint reports available capacity.

Answer

Request a service quota increase for provisioned model units so the endpoint is allocated instantly upon the create request.