Question

A machine learning team deploys a real-time SageMaker endpoint hosting a deep-learning image classification model built on a large convolutional neural network. During load testing on ml.c5.4xlarge CPU instances, per-request latency exceeds the 200 ms SLA, and CPU utilization is pinned at 100% even at moderate request volumes. The model uses matrix-heavy inference operations that parallelize well. What change should the team make to meet the latency requirement most effectively?

Accepted Answer

Switch the endpoint to GPU-accelerated instances such as ml.g5.xlarge, which are optimized for parallel matrix operations in deep-learning inference.

Answer

Convert the endpoint to a SageMaker Batch Transform job to spread the inference load over a longer processing window.

Answer

Add more ml.c5.4xlarge instances behind the endpoint using a target-tracking scaling policy on request count.

Answer

Migrate the model to a SageMaker Serverless Inference endpoint so capacity scales automatically per request.