Question

A data engineer builds a Spark Structured Streaming job in a Fabric notebook that reads events from an Eventstream and writes them into a Delta table in a lakehouse. The job runs continuously. Occasionally the cluster restarts due to maintenance, and the team requires that no events are lost and none are written twice after a restart. Which configuration ensures exactly-once processing semantics across restarts?

Accepted Answer

Specify a durable checkpoint location on OneLake in the writeStream options so offsets and state are persisted and recovered on restart.

Answer

Set the output mode to 'complete' and rely on Delta table overwrites to remove any duplicates on each micro-batch.

Answer

Increase the trigger interval to a large value so the stream processes fewer, larger batches and reduces the chance of overlap.

Answer

Enable auto-compaction on the Delta table so duplicate rows created after a restart are automatically merged away.