Question

A retail analytics team uses Cloud Data Fusion to build a batch pipeline that ingests daily CSV exports from partner SFTP servers into BigQuery. Occasionally a partner adds new columns or sends malformed rows, which currently causes the entire pipeline run to fail. The team wants the pipeline to continue processing valid records, route bad records to a separate location for later inspection, and avoid manual code changes for each new column. What is the most appropriate approach within Data Fusion?

Accepted Answer

Configure the pipeline's error handling to send failed records to an error collector plugin writing to a Cloud Storage error dataset, and enable a flexible schema in the Wrangler/parsing stage so unexpected columns do not abort the run

Answer

Set the pipeline to retry the entire run three times on failure, expecting the malformed rows to eventually succeed

Answer

Replace Data Fusion with a custom Dataflow pipeline because Data Fusion cannot handle schema drift or bad records

Answer

Enforce a strict fixed schema and fail the whole batch whenever any deviation occurs, then have an engineer manually edit the pipeline for every new column