Question

A data engineering team runs a nightly pipeline that has three sequential stages: a Glue ingestion job, a Glue transform job, and a Redshift COPY load. Each stage currently has its own CloudWatch alarm that sends an SNS notification on failure. On-call engineers complain that a single upstream failure produces a cascade of three separate pages, because when the ingestion job fails the downstream stages also alarm. The team wants a single actionable notification that fires only when the pipeline as a whole is in a failed state, while keeping the individual alarms for dashboards. What is the MOST operationally efficient way to achieve this?

Accepted Answer

Create a CloudWatch composite alarm that combines the three stage alarms with an alarm rule, and attach the SNS notification to the composite alarm instead of the individual alarms.

Answer

Delete the two downstream alarms and keep only the ingestion job alarm, since a failure there always blocks the rest of the pipeline.

Answer

Increase the evaluation period and datapoints-to-alarm on each individual alarm so they take longer to trigger and overlap less.

Answer

Route all three SNS notifications to an SQS queue and write a Lambda function that de-duplicates messages within a time window before paging on-call.