Question

A machine learning team has trained a new image classification model that will be deployed to a real-time SageMaker endpoint. They must handle a sustained throughput of about 200 requests per second while keeping p99 latency under 100 ms and minimizing cost. The team is unsure which instance type and count to provision and wants a data-driven recommendation before going to production. Which approach best meets these requirements with the least manual effort?

Accepted Answer

Run SageMaker Inference Recommender load tests against the model package to obtain instance type, instance count, and cost/latency recommendations.

Answer

Deploy the model to the largest available GPU instance type and rely on target-tracking autoscaling to reduce cost over time.

Answer

Deploy to a serverless inference endpoint and increase the memory configuration until latency targets are met.

Answer

Manually deploy the model to several candidate instance types one at a time and record throughput and latency using a custom script.