Question

Your team runs two independent Cloud Composer DAGs. The first DAG ingests raw sales events into BigQuery every hour. A separate downstream DAG builds hourly aggregate reports, but it must only run after the corresponding ingestion DAG run for that hour has fully completed. The teams want to keep the two DAGs separate for ownership reasons, and the downstream DAG should not start processing until the upstream data is confirmed present. What is the best way to coordinate these two DAGs?

Accepted Answer

Use an ExternalTaskSensor in the downstream DAG to wait for the successful completion of the upstream DAG's final task for the matching execution date.

Answer

Merge both DAGs into a single DAG so tasks run sequentially with normal task dependencies.

Answer

Add a fixed time_delta so the downstream DAG always starts 30 minutes after the upstream DAG's scheduled start.

Answer

Trigger the downstream DAG manually each hour after checking the upstream DAG's status in the Airflow UI.