Question

A data engineering team runs a nightly batch pipeline consisting of three sequential AWS Glue jobs: a raw ingestion job, a cleansing job, and an aggregation job. The team wants the pipeline to start automatically when a new file lands in an S3 bucket, and each Glue job should run only after the previous one completes successfully, with no code to manage between jobs. They prefer to keep the orchestration entirely within AWS Glue's native tooling rather than adding another service. Which approach best meets these requirements?

Accepted Answer

Create an AWS Glue workflow with an EventBridge trigger that starts on S3 object creation, followed by conditional triggers that fire each subsequent job on the SUCCEEDED state of the prior job.

Answer

Schedule all three Glue jobs with a cron-based Glue trigger set to the same time so they run together each night.

Answer

Use a single Glue job with three separate DynamicFrame transformations and enable job bookmarks to chain the steps automatically.

Answer

Configure an S3 event notification to invoke a Lambda function that calls StartJobRun three times in a loop for the three Glue jobs.