Question

Your team runs a critical streaming Dataflow pipeline that ingests payment events. Occasionally, a downstream API returns malformed responses, causing a specific worker exception that is logged but does not crash the job—instead, elements silently accumulate in a retry loop and processing slows. Operations wants to be automatically alerted the moment these exceptions begin appearing, before system lag grows large enough to breach the freshness SLA. Which approach best meets this requirement?

Accepted Answer

Create a log-based metric in Cloud Logging that counts occurrences of the specific worker exception, then configure a Cloud Monitoring alerting policy that fires when the metric exceeds zero over a short rolling window.

Answer

Enable Dataflow autoscaling with a higher maximum worker count so additional workers absorb the retry backlog before lag increases.

Answer

Set up a Cloud Monitoring alert only on the job's data freshness (system lag) metric with a high threshold to detect when the pipeline falls behind.

Answer

Schedule a Cloud Function every hour to query the Dataflow job status API and email the team if the job state is not RUNNING.