Question

A data engineering team needs to orchestrate a nightly pipeline: at 2:00 AM a Lambda function validates that new files have arrived in S3, then an AWS Glue ETL job transforms the data, and finally a second Lambda function sends a completion notification. The pipeline must run only on business days (Monday–Friday), must retry the Glue job up to 3 times on transient failures, and must give the team a visual view of which step failed. Which approach best meets these requirements with the least custom code?

Accepted Answer

Use an EventBridge Scheduler rule with a cron expression for weekdays at 2:00 AM to start an AWS Step Functions Standard workflow that sequences the Lambda validation, the Glue job (with a Retry block), and the notification Lambda.

Answer

Configure an AWS Glue workflow triggered by a scheduled cron trigger, and embed the validation and notification logic as Python shell jobs within the same workflow.

Answer

Schedule a Lambda function with an EventBridge rule that runs every night, and have that Lambda synchronously invoke the Glue job and handle retries and notifications in its own code.

Answer

Use an EventBridge Scheduler rule to start an AWS Step Functions Express workflow that chains the Lambda functions and Glue job, relying on the execution history for the visual view.