Question

A data engineer needs to run a one-time bulk load of several thousand historical Parquet files that were just uploaded to an S3 path into an existing Delta table. The load must be re-runnable safely (if the job is retried, previously loaded files should not be re-ingested), but the team does NOT want to configure a streaming checkpoint or maintain a continuously running stream. Which approach best meets these requirements?

Accepted Answer

Use the COPY INTO command targeting the Delta table with the source S3 path and Parquet format

Answer

Use spark.read.parquet() to load the files and then append them with a standard INSERT INTO

Answer

Configure Auto Loader with cloudFiles format and a continuous trigger

Answer

Use CREATE TABLE AS SELECT to rebuild the Delta table from the Parquet files on each run