Question

A team hosts 300 customer-specific fraud models behind a single SageMaker multi-model endpoint (MME) to save cost. Most models are invoked a few times per hour, but three high-traffic customers generate the majority of requests. Operators notice that these three heavily-used models occasionally see elevated p99 latency spikes that coincide with invocations of rarely-used models. What is the most likely cause and the best remedy?

Accepted Answer

MME evicts less-used models from instance memory to load newly requested ones; deploy the three high-traffic models on a separate dedicated endpoint while keeping the long tail on the MME.

Answer

The endpoint instance type is undersized; enable serverless inference for the MME to remove all memory constraints.

Answer

Target-tracking autoscaling is misconfigured; lower the InvocationsPerInstance target so more instances are added.

Answer

Data capture is throttling inference threads; disable data capture on the endpoint to eliminate the latency spikes.