Question

A retail company runs a critical Dataflow streaming pipeline that reads from Pub/Sub and writes aggregated sales metrics to BigQuery. During a regional service disruption, the job's workers become unresponsive and the pipeline stops processing. The on-call engineer must minimize data loss when the pipeline is restarted, and the team wants to reduce manual intervention for future incidents. Which approach best mitigates the impact of such failures?

Accepted Answer

Rely on Pub/Sub's default message retention so that when a new pipeline is launched against the same subscription, unacknowledged messages are redelivered and reprocessed.

Answer

Cancel the failed job and start a fresh job with a new Pub/Sub subscription so processing begins from the current messages only.

Answer

Increase the number of Dataflow workers manually after each outage to catch up on the backlog faster.

Answer

Switch the pipeline to batch mode so failed jobs can be retried from the beginning without message loss.