Question

A retail company deploys a trained image-classification model to thousands of in-store edge devices running on ARM-based hardware with limited CPU and memory. The ML team observes that inference latency on the devices is too high and the model binary consumes too much memory, causing occasional crashes. The team wants to reduce inference latency and the model's memory footprint on the target hardware without retraining the model or degrading accuracy. Which approach best meets these requirements?

Accepted Answer

Use SageMaker Neo to compile the trained model for the specific target ARM hardware and deploy the optimized runtime to the edge devices.

Answer

Deploy the model to a SageMaker real-time endpoint and have edge devices call it over the internet for every inference.

Answer

Increase the instance size used during training so the resulting model artifact is smaller and faster on the devices.

Answer

Enable SageMaker multi-model endpoints to host the model and reduce its memory footprint on the edge devices.