Question

Your team runs a Cloud Composer (Airflow) DAG that ingests partner data every 15 minutes. The upstream partner API is known to occasionally return transient 503 errors, and individual task retries usually recover the run. On-call engineers complain about alert fatigue because they receive a page every time a single task instance fails, even when the retry succeeds seconds later. Leadership still wants to be notified quickly when the pipeline is genuinely broken. Which approach best reduces noise while preserving fast awareness of real failures?

Accepted Answer

Configure task retries with exponential backoff, and create a Cloud Monitoring alerting policy that fires only when the DAG's failed-run metric exceeds a threshold over a rolling window (e.g., multiple consecutive failed runs).

Answer

Remove all task-level failure alerts and rely solely on the daily Airflow DAG summary email to review outcomes.

Answer

Increase the number of retries to 50 so tasks almost never report failure, eliminating the alerts entirely.

Answer

Set an on_failure_callback on every task that immediately pages on-call, and disable retries so failures surface faster.