Question

A data engineer is configuring an Auto Loader stream to ingest millions of small JSON files that land continuously in a cloud storage bucket. The team notices that the default file discovery approach requires repeatedly listing the entire directory, which becomes slow and expensive as the number of files grows. Which Auto Loader configuration should the engineer use to improve file detection efficiency at scale?

Accepted Answer

Set cloudFiles.useNotifications (file notification mode) so Auto Loader uses cloud queue/event notifications instead of directory listing

Answer

Switch from Auto Loader to COPY INTO, which automatically scales directory listing

Answer

Increase the trigger interval so the directory is listed less frequently

Answer

Disable schema inference to reduce the overhead of scanning files