Question

A retail analytics team runs a streaming pipeline on Dataflow that feeds BigQuery tables consumed by executive dashboards. The business requires that dashboard data be no more than 15 minutes stale, and the team must be notified before an SLA breach occurs so they can intervene. Occasionally, upstream Pub/Sub backpressure causes processing to slow down. Which monitoring approach best allows the team to detect and act on freshness degradation before the 15-minute SLA is violated?

Accepted Answer

Create a Cloud Monitoring alerting policy on the Dataflow job's system_lag and data_watermark_age metrics with a threshold below 15 minutes, and route notifications to an on-call channel.

Answer

Schedule a daily BigQuery query that counts rows loaded in the previous 24 hours and emails the team if the count is unexpectedly low.

Answer

Enable Dataflow autoscaling to the maximum worker count so the pipeline never falls behind, eliminating the need for alerting.

Answer

Configure a Cloud Logging sink that exports all pipeline logs to Cloud Storage for later analysis of freshness issues.