Question

A data engineer builds a Structured Streaming job that reads JSON files landing in cloud storage and writes them to a Delta table. During testing, the cluster is manually restarted several times. The engineer notices that after each restart the stream reprocesses all previously ingested files, creating duplicate records in the target table. What is the most likely cause and correct fix?

Accepted Answer

The writeStream query is missing a checkpointLocation option; specifying a durable checkpoint path lets the stream track processed offsets and resume without reprocessing.

Answer

The stream is missing a trigger interval; adding trigger(availableNow=True) will prevent duplicate ingestion after restarts.

Answer

The Delta table lacks a primary key constraint; adding one will automatically deduplicate records on write.

Answer

The readStream is using schema inference; disabling schema inference will stop the stream from rereading old files.