Question

A team hosts a fine-tuned open-weights LLM on an Amazon SageMaker real-time endpoint using inference components. Traffic follows a strong business-hours pattern: heavy load from 8 AM to 6 PM on weekdays and essentially zero traffic overnight and on weekends. During peak hours, occasional bursts cause 60-second cold-start-like delays as new model copies load. Leadership wants to eliminate idle GPU cost during off-hours while keeping burst latency acceptable during business hours. Which approach best meets both goals?

Accepted Answer

Configure application auto scaling on the inference component with a target-tracking policy on invocations per copy, set the minimum copy count to zero for scale-to-zero during idle periods, and add a scheduled scaling action that raises the minimum copy count before business hours to pre-warm capacity.

Answer

Switch the workload to Bedrock on-demand InvokeModel so AWS manages all scaling, and rely on exponential backoff to handle the overnight-to-morning traffic transition.

Answer

Purchase SageMaker provisioned capacity for the maximum expected peak throughput and keep it running 24/7 to guarantee that no cold-start delays ever occur.

Answer

Keep a fixed minimum of two model copies at all times and set a step-scaling policy that only scales up, never down, so capacity is always ready for bursts.