Question

A Fabric data engineer maintains a Lakehouse Delta table called Customers with 40 million rows. A daily pipeline extracts only rows that changed since the previous run (based on a LastModified column) into a staging Delta table. The engineer must apply these changes to the target Customers table so that existing customers are updated in place and brand-new customers are inserted, without rewriting the entire 40-million-row table each day. Which approach in a PySpark notebook best meets this requirement?

Accepted Answer

Use a Delta Lake MERGE operation matching on the customer key, with WHEN MATCHED THEN UPDATE and WHEN NOT MATCHED THEN INSERT clauses.

Answer

Overwrite the target Customers table with the union of the existing rows and the staging rows using DataFrame.write.mode('overwrite').

Answer

Append the staging rows to the target table using DataFrame.write.mode('append') and rely on Delta to deduplicate automatically.

Answer

Truncate the target Customers table and reload it entirely from the source system each day.