Question

A data engineering team receives new JSON files in a cloud storage location every few minutes, around 50,000 small files per day. They need a low-latency ingestion pipeline that automatically detects and processes newly arriving files without reprocessing old ones, and they want to avoid the overhead of listing the entire directory on each run. Which ingestion approach best meets these requirements?

Accepted Answer

Use Auto Loader with Structured Streaming and file notification mode to incrementally process new files as they arrive.

Answer

Run a scheduled COPY INTO command every few minutes that reads the entire storage directory each time.

Answer

Use a batch spark.read.json() job that overwrites the target table on every scheduled run.

Answer

Manually track processed file names in a control table and filter them out in each batch job.