Question

Your team is building a streaming pipeline where a Dataflow job must consume events from a Pub/Sub topic. The pipeline needs to autoscale based on backlog, provide low-latency processing, and let Dataflow control the rate at which it acknowledges messages to avoid overwhelming downstream sinks during traffic spikes. Which Pub/Sub consumption approach should you configure for the Dataflow job?

Accepted Answer

Use the Dataflow Pub/Sub I/O connector with a pull subscription so Dataflow manages flow control and acknowledgment

Answer

Create a push subscription that delivers messages to an HTTP endpoint fronting the Dataflow job

Answer

Create a BigQuery subscription so messages are written directly to BigQuery, then read them into Dataflow in batch

Answer

Export the topic to Cloud Storage with a Pub/Sub-to-GCS subscription and have Dataflow poll the bucket