Question

Your team uses Cloud Composer to orchestrate ingestion DAGs. The same DAG code must run in dev, staging, and prod environments, each pointing to different BigQuery datasets, Pub/Sub topics, and service accounts. Developers currently hardcode these values, causing errors when DAGs are promoted between environments. You want a maintainable approach that keeps a single DAG codebase while allowing per-environment configuration without editing the DAG files. What should you do?

Accepted Answer

Store environment-specific values in Airflow Variables and Connections that are set per Composer environment, and reference them in the DAG code.

Answer

Maintain three separate copies of each DAG, one per environment, and deploy the matching copy to each Composer environment.

Answer

Hardcode the production values in the DAG and use Airflow's built-in test mode to override them automatically in dev and staging.

Answer

Pass all environment values as XComs pushed from an upstream task at DAG runtime.