Question

A data engineer builds a Spark Structured Streaming job in a Fabric notebook that reads clickstream events from an Eventstream. The requirement is to continuously write these events into an existing Delta table in a lakehouse, but each incoming micro-batch may contain updates to previously written rows (identified by an event key), and the team wants those rows merged (upserted) rather than appended. Which approach should the engineer use in the streaming write?

Accepted Answer

Use writeStream with foreachBatch and, inside the function, perform a Delta Lake MERGE against the target table for each micro-batch

Answer

Use writeStream with outputMode("append") and let Delta automatically deduplicate on the key

Answer

Use writeStream with outputMode("complete") so the entire result table is recomputed and overwritten each trigger

Answer

Write each micro-batch to a temporary table with append mode and run a nightly batch job to merge the data