Question

Your team runs a nightly Dataflow batch pipeline that reads from Cloud Storage and writes aggregates to BigQuery. Currently an engineer manually launches the job each night from a Flex Template. Leadership wants the job to run automatically at 02:00 every day with no standing infrastructure, minimal operational overhead, and the ability to be triggered programmatically. Which approach best meets these requirements?

Accepted Answer

Create a Cloud Scheduler cron job at 02:00 that invokes a Cloud Run function (or the Dataflow templates API) to launch the Flex Template as a batch job.

Answer

Deploy a persistent Compute Engine VM running a cron daemon that calls gcloud dataflow flex-template run each night.

Answer

Convert the pipeline to a streaming Dataflow job that stays running continuously and emits results once per day.

Answer

Use a BigQuery scheduled query configured to run at 02:00 to trigger the Dataflow Flex Template.