Question

A data engineer maintains a Silver customer table in Delta Lake. Each micro-batch of change data contains one or more rows per customer_id, and the goal is to keep only the latest state per customer: update existing customers, insert new ones. Which approach correctly and idempotently applies these updates in a single command?

Accepted Answer

Use MERGE INTO with the target matched on customer_id, WHEN MATCHED THEN UPDATE SET *, and WHEN NOT MATCHED THEN INSERT *, after first deduplicating the source to one row per customer_id.

Answer

Use INSERT INTO to append all change rows, then rely on Delta to automatically discard older duplicate customer_id values.

Answer

Use MERGE INTO directly on the raw change batch without deduplication, matching on customer_id.

Answer

Use INSERT OVERWRITE on the entire target table with the incoming change batch each micro-batch.