Question

A retail company publishes order events to a single Pub/Sub topic. A Dataflow streaming pipeline must route events to different BigQuery tables based on the event's region, but the region value is only available inside the JSON message payload, not as a Pub/Sub attribute. The team wants to avoid deserializing the full payload twice and wants the routing decision to happen efficiently within the pipeline. What is the most appropriate approach?

Accepted Answer

Parse the payload once in a DoFn and use a tagged PCollection (multiple outputs) to split events by region, then write each tagged output to its corresponding BigQuery table

Answer

Create a separate Pub/Sub subscription with a filter expression for each region and run one Dataflow pipeline per subscription

Answer

Use a BigQuery subscription to write directly to a single table, then run scheduled queries to split rows into per-region tables

Answer

Republish each message to a region-specific Pub/Sub topic using a Cloud Function before Dataflow consumes it