Question

A media company needs to orchestrate an ML workflow that, once per hour, runs three independent SageMaker batch transform jobs against different regional datasets, then combines the outputs into a single ranking file only after all three jobs succeed. The team wants a serverless orchestration approach that runs the three jobs concurrently to minimize total runtime and that can pass each job's output location downstream to a final aggregation Lambda. Which orchestration design best meets these requirements?

Accepted Answer

Use an AWS Step Functions state machine with a Parallel state containing three branches that each invoke a batch transform job, then transition to a Lambda task that aggregates the outputs after all branches complete.

Answer

Use a single SageMaker Pipelines TransformStep with a loop that processes each regional dataset sequentially, followed by a ProcessingStep to aggregate results.

Answer

Configure an EventBridge scheduled rule that triggers three separate Lambda functions, each launching a batch transform job, and have the third Lambda perform aggregation immediately after it starts its job.

Answer

Create three independent SageMaker Pipelines on an hourly schedule and use an S3 event notification on the last-written output file to trigger the aggregation Lambda.