Question

A data engineer builds an Auto Loader stream that ingests JSON files from a cloud storage path into a Bronze Delta table. The pipeline runs every hour using Trigger.AvailableNow. After a job restart, the engineer notices that previously ingested files are NOT being reprocessed, even though the source directory still contains all historical files. Which mechanism is responsible for ensuring each file is processed exactly once across restarts?

Accepted Answer

Auto Loader's checkpoint location, which persists metadata about which files have already been processed

Answer

The Delta transaction log of the target table, which deduplicates rows on write

Answer

The cloudFiles.schemaLocation option, which caches file contents between runs

Answer

The VACUUM command, which removes already-ingested files from the source directory