Question

A data engineer configures an Auto Loader stream to ingest millions of new files that arrive daily into a large cloud storage bucket. The initial deployment uses the default directory listing mode, but as the number of files grows, the team notices that each micro-batch spends a long time simply discovering new files, increasing latency and cloud API costs. Which change should the engineer make to improve file-discovery efficiency at this scale?

Accepted Answer

Switch Auto Loader to file notification mode by setting cloudFiles.useNotifications to true so new files are detected via cloud queue events instead of listing the directory

Answer

Replace Auto Loader with a COPY INTO command that reloads the entire directory on each run

Answer

Increase the cluster size so directory listing completes faster during each micro-batch

Answer

Set cloudFiles.inferColumnTypes to false to reduce the overhead of scanning file contents