Question

A healthcare analytics company runs a streaming Dataflow pipeline that reads patient event data from Pub/Sub, transforms it, and writes to BigQuery. Compliance requires that if any record fails validation (e.g., malformed PHI fields), it must not be silently dropped and must be available for later inspection without halting the pipeline. Which design best satisfies this reliability and fidelity requirement?

Accepted Answer

Route validation failures to a dead-letter destination (e.g., a separate BigQuery table or Cloud Storage bucket) using a tagged output while continuing to process valid records.

Answer

Configure the pipeline to throw an exception on the first invalid record so the job fails fast and no bad data is written.

Answer

Enable Dataflow autoscaling and rely on the built-in retry policy to eventually process the malformed records successfully.

Answer

Filter out invalid records with a Filter transform so only clean data reaches BigQuery, keeping the pipeline simple.