Question

A retail analytics team runs a Cloud Composer DAG every hour to load transactional data into BigQuery. Occasionally an upstream API is slow, causing a DAG run to take over 90 minutes. When this happens, multiple overlapping DAG runs start, all writing to the same partitioned staging table, which produces duplicate rows and downstream reconciliation failures. The team wants to prevent overlapping runs of this specific DAG while keeping the hourly schedule and without disabling other DAGs in the environment. What is the most appropriate change?

Accepted Answer

Set max_active_runs=1 on the DAG so only one run executes at a time and later scheduled runs queue until the current run completes.

Answer

Increase the environment's worker count and Airflow parallelism setting so all overlapping runs finish faster.

Answer

Enable catchup=True on the DAG so Airflow serializes the backlog of runs into a single ordered sequence.

Answer

Set the DAG's schedule_interval to '@once' and trigger it manually after confirming the API is responsive.