Question

A retail company is migrating a nightly on-premises ETL job to Google Cloud. During the migration cutover period, both the legacy on-premises pipeline and the new Dataflow pipeline will run in parallel against the same source data for two weeks to build confidence before decommissioning the old system. The data engineering team must prove that the new pipeline produces identical aggregated results to the legacy system before switching over. Which approach best validates output fidelity during this parallel-run period?

Accepted Answer

Load both pipeline outputs into separate BigQuery tables and run scheduled reconciliation queries that compare row counts, checksums, and aggregate totals, flagging any discrepancies for investigation.

Answer

Rely on Dataflow's built-in exactly-once processing guarantees, which ensure the new pipeline output automatically matches the legacy system.

Answer

Decommission the legacy pipeline immediately after the first successful Dataflow run and monitor downstream dashboards for anomalies.

Answer

Enable Cloud Monitoring alerts on the Dataflow job's system lag and element throughput metrics to confirm the outputs are correct.