Question

A Dataflow streaming pipeline aggregates ad-impression counts per publisher using a GroupByKey followed by a Sum combiner. A handful of very large publishers generate the vast majority of events, and monitoring shows severe data skew: a few workers are overloaded processing these 'hot keys' while others sit idle, causing backlog growth. You must reduce processing latency without changing the final aggregation results. What is the most effective change?

Accepted Answer

Apply Combine.PerKey with the .withHotKeyFanout() setting so hot keys are partially combined across multiple intermediate workers before the final merge.

Answer

Replace the streaming pipeline with a batch pipeline that reprocesses the data hourly to avoid real-time skew.

Answer

Increase the maximum number of workers so more machines are available to pick up the hot-key work.

Answer

Switch from fixed windows to global windows so the combiner has more time to distribute the load evenly.