Question

A data engineer ingests daily CSV extracts from a partner into a Fabric lakehouse using a PySpark notebook. Over time, the partner occasionally adds new columns to later files, but historical files keep the older, narrower schema. When the engineer reads all files at once with spark.read.csv() and a wildcard path, some rows lose data from the newer columns. The engineer must combine all files into a single Delta table that preserves every column that has ever appeared, filling missing values with nulls where a file lacked a column. Which approach best handles this schema drift?

Accepted Answer

Read each file into a DataFrame, then combine them using unionByName with allowMissingColumns=True before writing the result to the Delta table.

Answer

Read all files with a single spark.read.csv() call using inferSchema=True and write directly to the Delta table.

Answer

Read each file into a DataFrame and combine them with the standard union() operator, which aligns columns automatically by name.

Answer

Read only the newest file to capture the full schema, then append the older files using overwrite mode on the Delta table.