Question

A data engineering team runs several Apache Beam pipelines on Dataflow. Analysts frequently request the same pipeline with different input paths, output tables, and window sizes. Currently, launching a job requires a developer to build the pipeline locally with the SDK and pass parameters at compile time, which creates a bottleneck. The team wants to let non-developers launch parameterized jobs from the Console, gcloud, or a REST call without rebuilding code, while keeping the pipeline logic packaged and version-controlled. Which approach best meets these flexibility and reusability requirements?

Accepted Answer

Package each pipeline as a Dataflow Flex Template stored in Artifact Registry and Cloud Storage, exposing runtime parameters that callers supply at launch time.

Answer

Give analysts the source repository and IAM access so they can compile and run the Beam pipeline from their own workstations.

Answer

Schedule the pipeline with a fixed configuration in Cloud Composer and have analysts edit the DAG file whenever parameters change.

Answer

Convert the pipelines to Dataproc Spark jobs so they can be submitted from the Console with cluster properties.