Question

A data engineering team receives daily drops of thousands of small JSON files into a cloud storage directory. They need a solution that automatically discovers and ingests only new files on each scheduled run, without manually tracking which files have already been loaded, and that requires minimal SQL or Python code for a Delta target. Given these requirements, which approach best fits a repeatable, incremental ingestion pipeline that scales as file counts grow over time?

Accepted Answer

Use Auto Loader (cloudFiles) reading from the directory into a Delta table, relying on its checkpoint to track processed files across runs

Answer

Write a Spark batch job that reads the entire directory with spark.read.json each run and overwrites the target table

Answer

Manually maintain a control table listing loaded filenames and filter them out with a custom join before each DataFrame write

Answer

Mount the storage and use dbutils.fs.ls to enumerate files, then loop and load each file individually with INSERT INTO