Question

Your team uses Cloud Data Fusion to build a batch ETL pipeline that reads CSV files from Cloud Storage, applies several transformations, and writes to BigQuery. Before deploying the pipeline to production, business analysts report that certain output records have unexpected null values. You need to inspect how each transformation stage processes a small sample of the source data without incurring the cost of provisioning a full Dataproc cluster or writing to the production BigQuery table. What should you do?

Accepted Answer

Run the pipeline in Preview mode within the Data Fusion Studio to view intermediate stage outputs against sample data before deployment.

Answer

Deploy the pipeline and enable Dataproc job logging in Cloud Logging, then trace null values by reviewing the driver logs.

Answer

Add a Wrangler directive to filter out null records and redeploy the pipeline to production immediately.

Answer

Export the pipeline as a JSON configuration and manually simulate each transformation in a local Spark shell.