Question

Your team runs a Cloud Composer (Airflow) environment that orchestrates dozens of DAGs. Several DAGs each spawn many tasks that submit BigQuery load jobs against the same reservation. During peak hours, you notice a spike in failures caused by exceeding the concurrent load job quota, and downstream tasks intermittently fail. You want a repeatable, centralized way to cap the number of these tasks that run simultaneously across all DAGs without rewriting task logic or reducing overall DAG concurrency for unrelated tasks. What should you do?

Accepted Answer

Create an Airflow pool with a fixed number of slots and assign the BigQuery load tasks to that pool.

Answer

Lower the environment-wide parallelism (core.parallelism) setting so fewer tasks run at once.

Answer

Add exponential-backoff retries with a high retry count to each BigQuery load task.

Answer

Set max_active_runs=1 on every DAG that submits BigQuery load jobs.