Question

An analytics engineer ingests daily Parquet files into a Fabric lakehouse Delta table using a Spark notebook with df.write.format('delta').mode('append').saveAsTable('sales'). Today's file added a new column 'discount_code' that did not exist in prior files, and the append fails with a schema mismatch error. The requirement is to keep loading new files automatically and allow the table schema to grow when source files add new columns, without manually redefining the table each day. What should the engineer do?

Accepted Answer

Add the option 'mergeSchema' set to 'true' on the write so Delta evolves the table schema to include the new column.

Answer

Change the write mode from 'append' to 'overwrite' so the entire table is rewritten with the new schema each day.

Answer

Drop the 'discount_code' column from the incoming DataFrame before writing so it matches the existing table schema.

Answer

Set the option 'overwriteSchema' to 'true' on the append so the schema is replaced without losing prior rows.