Question

A data engineer receives a one-time delivery of exactly 12 large Parquet files in a cloud storage folder representing a historical backfill. The files will never change and no new files will arrive in this folder. The engineer needs to load them into an existing Delta table as a single batch operation and wants the simplest approach that guarantees each file is loaded only once even if the command is accidentally re-run. Which approach best fits this requirement?

Accepted Answer

Use COPY INTO to load the Parquet files into the Delta table, relying on its built-in tracking of already-ingested files.

Answer

Configure Auto Loader with cloudFiles and a checkpoint location running in continuous streaming mode.

Answer

Read the files with spark.read.parquet and append them using a plain INSERT INTO each time the job runs.

Answer

Use Auto Loader with directory listing mode and Trigger.AvailableNow, adding a new checkpoint each run.