Question

A retail analytics team ingests daily CSV and JSON files from dozens of partner systems whose schemas change frequently and without notice. They want a design that lets them re-run transformations against raw source data if a downstream schema mapping proves incorrect, while keeping BigQuery as the serving layer. Which design best supports flexibility and reprocessing?

Accepted Answer

Land raw files unaltered in a Cloud Storage staging bucket, apply transformations via a pipeline into curated BigQuery tables, and retain the raw zone so any file can be reprocessed later.

Answer

Load each file directly into partitioned BigQuery tables with autodetect schema and delete source files after load to save storage costs.

Answer

Configure partners to POST records directly into BigQuery through the streaming insert API so no intermediate storage is needed.

Answer

Transform files in place on the on-premises servers and only send finalized, schema-conformant Avro files to BigQuery.