Question

A retail analytics team runs a nightly workflow that must execute in a strict sequence: first a Dataproc Spark job that cleanses raw sales data in Cloud Storage, then a BigQuery load job that ingests the cleansed output, and finally a data-quality validation query. Each step must only start after the previous one succeeds, and failures must trigger automatic retries and email alerts to the on-call engineer. The team wants a managed, code-based way to define these dependencies and schedule the run each night. Which approach best meets these requirements?

Accepted Answer

Author a Cloud Composer DAG using DataprocSubmitJobOperator, BigQueryInsertJobOperator, and downstream task dependencies with retries and email_on_failure configured, scheduled to run nightly.

Answer

Configure a Cloud Scheduler job that publishes to a Pub/Sub topic, and have each pipeline stage subscribe and trigger the next stage via Cloud Functions.

Answer

Use a single Dataflow streaming pipeline that reads the raw data, performs cleansing, writes to BigQuery, and runs validation as sequential transforms.

Answer

Create separate Cloud Scheduler cron jobs for each step, spacing them apart in time so each has enough of a window to finish before the next begins.