Question

A data engineer is building a Structured Streaming job that reads change events from a source and must apply them as upserts (MERGE) into a Delta table for each incoming micro-batch. Which approach correctly implements per-micro-batch custom merge logic in the streaming write?

Accepted Answer

Use writeStream with .foreachBatch(...) and inside the function run a Delta Lake MERGE against the target table

Answer

Use writeStream with outputMode('complete') so the full result table is recomputed and merged automatically each trigger

Answer

Use writeStream with .format('delta').outputMode('append'), which performs automatic upserts on primary keys

Answer

Use writeStream with .trigger(availableNow=True) to enable MERGE semantics on the sink