Question

You deploy a new Cloud Composer (Apache Airflow) DAG with a daily schedule and a start_date set three months in the past. After you unpause the DAG, dozens of DAG runs trigger simultaneously, overwhelming your BigQuery slots and downstream APIs. You only want the DAG to run going forward, on its normal daily cadence, without executing all the historical intervals. What is the most appropriate way to prevent this behavior for future deployments?

Accepted Answer

Set catchup=False on the DAG so Airflow only schedules the most recent interval instead of backfilling all missed intervals since start_date.

Answer

Increase the Composer environment's worker count and set max_active_runs to a high value so all historical runs complete faster.

Answer

Change the start_date to datetime.now() dynamically in the DAG file so no past intervals ever exist.

Answer

Set the schedule_interval to None and trigger the DAG manually each day using a Cloud Scheduler HTTP call.