Question

A data engineer runs a nightly job that uses COPY INTO to load newly arrived CSV files from a cloud storage location into a Delta table. Due to a scheduling misconfiguration, the same job occasionally runs twice on the same night, pointing at the same source directory. The engineer notices that despite the duplicate runs, no duplicate rows appear in the target table. What is the primary reason COPY INTO avoids reloading the same data?

Accepted Answer

COPY INTO tracks which files have already been ingested and skips files it has previously loaded, making it idempotent on repeated runs.

Answer

COPY INTO automatically runs a MERGE against a primary key to deduplicate rows on every execution.

Answer

COPY INTO enables Delta's VACUUM process, which removes duplicate rows after each load.

Answer

COPY INTO overwrites the entire target table on every run, so duplicates cannot accumulate.