Question

Your data engineering team maintains several Dataflow pipelines whose source code lives in a Git repository. Currently, engineers manually build templates and deploy them from their laptops, which has led to inconsistent deployments and undetected regressions reaching production. You are asked to implement a CI/CD process that automatically runs unit and integration tests on every merge request, and only promotes validated pipeline artifacts to the production project after approval. Which approach best meets these requirements?

Accepted Answer

Configure Cloud Build triggers on the repository to run the Beam test suite (DirectRunner for unit tests, TestDataflowRunner for integration tests), build a Dataflow Flex Template, store it in Artifact Registry/Cloud Storage, and promote it to the production project via a gated build step.

Answer

Schedule a Cloud Composer DAG that periodically pulls the latest code, runs the pipeline in production, and rolls back if the job fails.

Answer

Have each engineer run pytest locally before pushing, then use a shared service account so everyone deploys templates directly to the production project.

Answer

Enable Dataflow's built-in continuous deployment feature to auto-deploy the newest commit whenever the main branch changes, skipping separate test stages.