Question

A data engineering team orchestrates a nightly batch pipeline using AWS Step Functions. The workflow runs three sequential AWS Glue jobs: raw ingestion, transformation, and aggregation. The transformation job occasionally fails due to transient throttling from a downstream service, and when it fails the entire pipeline aborts and must be manually restarted from the beginning, re-running the expensive ingestion job. The team wants the pipeline to automatically recover from transient failures without re-running successful upstream steps. What is the MOST operationally efficient way to achieve this?

Accepted Answer

Add a Retry field with exponential backoff to the transformation task state in the Step Functions state machine, targeting the relevant Glue error types.

Answer

Wrap all three Glue jobs into a single Glue job with internal try/except logic so failures are handled within one job run.

Answer

Schedule the entire state machine to run every 15 minutes via EventBridge so a failed run is quickly retried automatically.

Answer

Move the orchestration to a Lambda function that invokes each Glue job sequentially and sleeps between retries on failure.