Question

A retail company runs a critical streaming Dataflow pipeline that ingests order events from Pub/Sub and writes to BigQuery. During a flash sale, business users report that dashboards are showing data that is roughly 40 minutes stale, even though the pipeline shows no errors and has not stopped. The on-call data engineer needs to quickly confirm whether the pipeline is falling behind on processing before deciding whether to intervene. Which Dataflow monitoring signal should they check first to diagnose the staleness?

Accepted Answer

The data freshness and system latency metrics in the Dataflow job graph, along with the Pub/Sub subscription backlog (unacknowledged message count)

Answer

The BigQuery job history to confirm that load jobs completed successfully during the sale window

Answer

The Cloud Billing report to check whether the streaming engine cost spiked during the sale

Answer

The worker CPU utilization graph to confirm that vCPUs are not idle