Question

A retail company runs a streaming Dataflow pipeline that reads clickstream events from Pub/Sub, parses each JSON message, and writes structured rows to BigQuery. Occasionally, upstream systems emit malformed JSON or messages with schema violations. Currently, a single bad message causes the pipeline to throw an unhandled exception, stalling processing and creating a backlog. The data engineering team must ensure the pipeline keeps processing valid events while preserving the invalid ones for later investigation, without losing any data. What is the best design?

Accepted Answer

Wrap the parsing logic in a try/catch and route failed records to a dead-letter Pub/Sub topic or Cloud Storage bucket using a tagged (side) output, while valid records continue to BigQuery.

Answer

Enable Dataflow autoscaling and increase the maximum number of workers so the pipeline can absorb the backlog caused by malformed messages.

Answer

Configure the pipeline to silently drop any message that fails parsing so throughput is never interrupted.

Answer

Add a Pub/Sub schema definition to the topic so that malformed messages are rejected at publish time and never reach the pipeline.