Question

A data engineering team ingests new JSON order files that land in a cloud storage location throughout the day. They want to run a scheduled job every hour that processes only files that have arrived since the last run, then shuts down the cluster to save cost. They want to reuse Auto Loader's file tracking so no file is processed twice across runs. Which approach meets these requirements?

Accepted Answer

Use an Auto Loader (cloudFiles) readStream with Trigger.AvailableNow and a checkpoint location, scheduled as an hourly job

Answer

Use an Auto Loader readStream with Trigger.ProcessingTime='1 hour' running as a continuous always-on stream

Answer

Use spark.read.json() in a batch job and manually filter files by modification timestamp each hour

Answer

Use COPY INTO with FORCE = true to reload the source directory on every scheduled run