Question

A retail company needs to run nightly batch scoring jobs across a variable list of regional datasets (some nights 5 regions, other nights 30). Each region requires its own SageMaker batch transform job, but the datasets are independent and can be processed concurrently. The team wants a serverless orchestration approach that dynamically fans out one batch transform job per region without hardcoding the number of jobs, and that requires minimal custom code. Which approach best meets these requirements?

Accepted Answer

Use an AWS Step Functions state machine with a Map state that iterates over the region list and starts a SageMaker CreateTransformJob task for each item.

Answer

Create a single SageMaker batch transform job that reads all regional datasets from one S3 prefix and relies on multiple instances for parallelism.

Answer

Build a SageMaker Pipeline with a fixed number of TransformStep definitions, one per region, and update the pipeline definition each night.

Answer

Trigger a Lambda function per region via an EventBridge scheduled rule, and have each Lambda synchronously wait for its batch transform job to complete.