Question

Your team runs a Dataflow streaming pipeline that reads from Pub/Sub, performs enrichment lookups, and writes to BigQuery. During a marketing campaign, the Pub/Sub subscription backlog grows steadily and end-to-end latency climbs from seconds to several minutes. Monitoring shows worker CPU is only moderately utilized, but the number of workers has not increased. What is the most likely cause and appropriate fix?

Accepted Answer

Streaming Engine is disabled and the job uses a fixed number of workers; enable Streaming Engine and Horizontal Autoscaling so the pipeline can add workers based on backlog.

Answer

The BigQuery streaming insert quota was exceeded; migrate the sink to batch loads to remove the bottleneck.

Answer

The Pub/Sub topic has too few partitions; increase the topic partition count to allow parallel consumption.

Answer

The pipeline uses event-time windowing with an overly long allowed lateness; reduce the allowed lateness to lower latency.