Question

A financial services company runs a real-time fraud detection model on a SageMaker endpoint. The ML team has trained a new model version and wants to evaluate how it would perform on live production traffic without exposing any of its predictions to end users or affecting the current model's responses. They need to compare the new model's inference behavior against the production model using real request patterns before committing to a full rollout. Which approach best meets these requirements?

Accepted Answer

Configure a SageMaker shadow test that deploys the new model as a shadow variant, replicating production traffic to it while only the production variant's responses are returned to callers.

Answer

Deploy the new model as a second production variant with a 50/50 traffic split and monitor CloudWatch metrics for both variants.

Answer

Use a blue/green deployment with canary traffic shifting to gradually route a small percentage of live requests to the new model.

Answer

Run a SageMaker batch transform job on a historical dataset and compare the new model's outputs against the logged production predictions.