Question

A logistics company receives daily inventory files (about 5 GB each) dropped into a Cloud Storage bucket every night at 2 AM. The data engineering team must transform and load these files into BigQuery so that analysts have fresh data by 6 AM. There is no requirement for sub-hourly latency, and the files arrive as a predictable, bounded set each night. Which pipeline design best fits these requirements while minimizing operational cost?

Accepted Answer

Build a batch Dataflow pipeline triggered by Cloud Composer once the nightly files have landed, processing the bounded dataset and writing to BigQuery.

Answer

Build a streaming Dataflow pipeline reading from a Pub/Sub notification topic and keep it running 24/7 to process each file as it lands.

Answer

Configure a Pub/Sub-to-BigQuery direct subscription so the files are written to BigQuery automatically without transformation.

Answer

Use a streaming Dataflow pipeline with fixed 5-minute windows to continuously aggregate the files throughout the day.