Question

A data engineering team runs a nightly AWS Glue ETL job that populates a reporting table. Recently, the job has intermittently failed, but the team only discovers the failures the next morning when business users complain about missing dashboard data. The team wants to be notified automatically within minutes whenever the Glue job enters a failed state, with the least operational overhead. Which approach should they implement?

Accepted Answer

Create an Amazon EventBridge rule that matches the Glue Job State Change event for the FAILED state and targets an Amazon SNS topic that emails the team.

Answer

Enable Glue job bookmarks and configure the job to retry three times, then write failure messages to an S3 bucket that the team reviews each morning.

Answer

Schedule a Lambda function to run every hour that queries the Glue GetJobRuns API and sends an email if any run failed in the past 24 hours.

Answer

Configure Amazon Athena to query the Glue job's CloudWatch logs on a nightly schedule and export the results to a QuickSight dashboard.