Question

A financial services company ingests trade events through Cloud Pub/Sub into a Dataflow streaming pipeline. Auditors require that events for each trading account are processed in the exact order they were published, and that no event is dropped or duplicated in the final BigQuery ledger table. Traffic is bursty and can exceed 50,000 messages per second across thousands of accounts. Which design best meets these fidelity requirements?

Accepted Answer

Publish with an ordering key set to the trading account ID, enable message ordering on the subscription, and configure the Dataflow pipeline to write to BigQuery with exactly-once processing.

Answer

Publish messages without ordering keys and rely on Pub/Sub timestamps, then sort events in a downstream batch job that runs hourly over the BigQuery table.

Answer

Use a single-partition Pub/Sub topic to guarantee global ordering and process messages sequentially in a single-worker Dataflow job.

Answer

Enable at-least-once delivery only and deduplicate in BigQuery using scheduled MERGE queries, accepting that ordering may vary per account.