Question

A data engineering team runs a nightly AWS Glue ETL job that normally completes in about 25 minutes. Over the past two weeks, the job's runtime has gradually crept up to over 90 minutes, though it still succeeds. Leadership wants proactive alerting when future jobs deviate significantly from their historical baseline, without hard-coding a fixed threshold that must be manually adjusted as data volume grows. Which approach best meets this requirement?

Accepted Answer

Create a CloudWatch alarm on the Glue job's execution duration metric using anomaly detection, and configure the alarm to trigger an SNS notification when the metric falls outside the expected band.

Answer

Create a static CloudWatch alarm on the glue.driver.aggregate.elapsedTime metric with a fixed threshold of 30 minutes and update it manually each quarter.

Answer

Enable Glue job bookmarks so the job processes less data and finishes faster, eliminating the need for duration monitoring.

Answer

Configure a CloudWatch Logs metric filter on the Glue job logs to count log lines and alarm when the count exceeds a fixed value.