Question

A data engineer uses Auto Loader to incrementally ingest JSON files from a cloud storage location into a Bronze Delta table. The upstream system occasionally adds new fields to the JSON files. The engineer wants the pipeline to automatically detect and add these new columns to the target table without the stream failing, and to reprocess the affected data. Which configuration should the engineer use?

Accepted Answer

Set cloudFiles.schemaEvolutionMode to "addNewColumns" (the default) and provide a schemaLocation so Auto Loader tracks the evolving schema.

Answer

Set cloudFiles.schemaEvolutionMode to "none" so the stream ignores unexpected columns and continues running.

Answer

Use COPY INTO with the FORMAT_OPTIONS('mergeSchema' = 'false') setting to avoid schema conflicts.

Answer

Disable schema inference by hardcoding a fixed StructType schema so no new columns can ever appear.