Question

Your team runs about 300 DAGs in a Cloud Composer 2 environment. Recently, the Airflow scheduler has become sluggish: tasks are queued but slow to start, and the DAGs list in the UI takes a long time to refresh. On investigation, you find that several DAG files execute heavy top-level code — including API calls to fetch configuration and expensive pandas transformations — every time the file is parsed. What is the most effective change to improve scheduler performance?

Accepted Answer

Move the heavy top-level code into task callables (inside operators/PythonOperator functions) so it runs only at task execution time, not on every DAG parse.

Answer

Increase the Composer environment's worker node count and worker concurrency so more tasks can run in parallel.

Answer

Set a very short scheduler `dag_dir_list_interval` so the scheduler re-parses DAG files more frequently and stays up to date.

Answer

Combine all 300 DAGs into a single large DAG file to reduce the number of files the scheduler must scan.