Question

A retail analytics team runs a streaming Dataflow pipeline that must join two Pub/Sub streams: an 'orders' stream and a 'payments' stream. An order and its matching payment share an order_id but can arrive up to 10 minutes apart and out of order. The team needs to correlate each order with its payment to compute settled revenue, emitting a result once both events for an order_id are seen (or marking the order unsettled if no payment arrives within the window). Which pipeline design best meets these requirements?

Accepted Answer

Key both streams by order_id, apply a matching event-time windowing strategy (e.g., a fixed or sliding window covering the 10-minute skew) with allowed lateness, and use CoGroupByKey to co-group orders and payments per key before emitting settled/unsettled results.

Answer

Apply an event-time session window with a 10-minute gap duration to each stream independently, then use a side input to look up payments while processing orders.

Answer

Write both streams directly to BigQuery via Pub/Sub subscriptions and perform the join with a scheduled SQL query that runs every 10 minutes.

Answer

Use a global window with a GroupByKey on order_id and no triggers so all matching records accumulate indefinitely before the join is emitted.