Question

A retail company runs a streaming Dataflow pipeline that ingests order events from a single Pub/Sub topic. The business now wants to consolidate events from five additional regional Pub/Sub topics into the same processing logic, applying identical transformations and writing to one BigQuery table. The team wants to minimize operational overhead and avoid running six separate pipelines, while retaining the ability to identify each event's source region downstream. What is the best approach?

Accepted Answer

Create one Pub/Sub subscription per topic, read all subscriptions in a single Dataflow pipeline using multiple PubsubIO reads, tag each element with its source, flatten the collections, and apply the shared transforms.

Answer

Deploy six independent Dataflow jobs, one per topic, each writing to the same BigQuery table, and add a static region label in each job's configuration.

Answer

Configure Pub/Sub to fan-in all six topics into a single topic using topic-to-topic forwarding, then read the consolidated topic with one subscription in Dataflow.

Answer

Export all six topics to Cloud Storage and run a scheduled batch Dataflow job that unions the files before loading into BigQuery.