Question

A data engineering team runs a nightly AWS Glue ETL job triggered by an Amazon EventBridge cron rule at 02:00 UTC. Due to a multi-hour AWS maintenance window in a downstream dependency, the source data was not available for three consecutive nights, so the job ran but produced empty output files that were written to the curated S3 zone. Analysts querying the data with Athena now see gaps and empty partitions. The team wants to prevent the pipeline from writing output when the expected upstream data has not landed, while keeping the automated schedule. Which approach BEST addresses this requirement?

Accepted Answer

Add a preliminary validation step (for example, a Lambda function or an initial Glue job) that checks for the presence and minimum record count of the source data, and only proceed to the transformation job if the check passes.

Answer

Increase the EventBridge cron schedule frequency to run the Glue job every hour so that it eventually picks up the source data when it lands.

Answer

Enable Glue job bookmarks so the job automatically skips runs where no new source data is present.

Answer

Configure a CloudWatch alarm on the Glue job's completion metric to notify the team after each empty run so they can manually delete the empty partitions.