Question

A retail company runs a Dataflow streaming pipeline that reads transaction events from Pub/Sub, enriches them with a REST call to an external inventory API, and writes results to BigQuery. During a promotional event, the external API begins returning intermittent HTTP 503 errors. The team observes that some elements fail permanently and cause the pipeline to stall, while transient failures should be retried without dropping data. What is the most appropriate way to build robust error handling into this transformation step?

Accepted Answer

Use a DoFn with tagged outputs (multiple outputs) to route successfully enriched records to the main output and failed records to a dead-letter output that is written to a separate BigQuery table or Pub/Sub topic for later reprocessing.

Answer

Wrap the API call in a try/except that swallows all exceptions and returns null so the pipeline never stalls, then filter out null records before the BigQuery write.

Answer

Set the pipeline to cancel and restart automatically whenever the API returns a 503, relying on Pub/Sub redelivery to reprocess the entire backlog.

Answer

Increase the number of workers via a fixed high worker count so the pipeline has enough parallelism to overcome the API errors through brute force retries.