Question

A data science team needs to orchestrate an ML workflow that spans multiple AWS services: an AWS Glue job for preprocessing, a SageMaker training job, a Lambda function to evaluate metrics, and conditional publishing to an SNS topic. The workflow must include automatic retries with exponential backoff on transient failures, branching logic based on the evaluation output, and a visual audit trail of each execution's state transitions for compliance auditors. Which orchestration approach best meets these requirements with the least custom code?

Accepted Answer

Use AWS Step Functions with a state machine that integrates Glue, SageMaker, Lambda, and SNS via optimized service integrations, using Retry and Choice states.

Answer

Use SageMaker Pipelines exclusively, adding a Lambda step and a callback step to trigger the Glue job and SNS notification.

Answer

Write a Python orchestration script running on an EC2 instance that calls each service's API sequentially and logs transitions to CloudWatch.

Answer

Use Amazon EventBridge rules to chain each service invocation and rely on service-native retry settings for failure handling.