Question

A retail company ingests order events into Pub/Sub, where an upstream publisher occasionally retries publishes, causing duplicate messages with the same application-level order_id but different Pub/Sub message IDs. A Dataflow streaming pipeline aggregates order totals in 10-minute fixed windows and writes to BigQuery. The team observes inflated revenue totals due to duplicate orders. They cannot change the publisher. Which approach most reliably eliminates the duplicates based on order_id within the pipeline?

Accepted Answer

Apply a Dataflow Deduplicate transform keyed on the order_id attribute with a time-bounded deduplication window sized to cover the expected retry interval.

Answer

Enable Pub/Sub exactly-once delivery on the subscription so duplicate order events are automatically removed before reaching Dataflow.

Answer

Increase the Pub/Sub message retention duration and rely on Dataflow's built-in checkpointing to discard repeated messages.

Answer

Switch the windowing strategy from fixed windows to session windows so that duplicate orders collapse into the same session.