Question

A retail company operates multiple microservices that each publish order events to a shared Pub/Sub topic consumed by a Dataflow streaming pipeline. Recently, one service began publishing malformed JSON with missing required fields, causing repeated deserialization failures downstream. The data engineering team wants to prevent structurally invalid messages from ever entering the topic, rejecting them at publish time before they reach any subscriber. What is the most effective way to enforce this?

Accepted Answer

Attach a Pub/Sub schema (Avro or Protocol Buffers) to the topic so that messages violating the schema are rejected when publishers attempt to publish them.

Answer

Add a dead-letter topic to the subscription so invalid messages are routed away after the Dataflow pipeline fails to process them.

Answer

Enable Pub/Sub message ordering on the topic to guarantee that only well-formed messages are delivered in sequence.

Answer

Configure a Dataflow filter transform early in the pipeline that discards messages missing required fields.