Data Ingestion and Loading

A data engineering team ingests new JSON order files that land in a cloud storage location throughout the day. They want to run a scheduled job every hour that processes only files that have arrived since the last run, then shuts down the cluster to save cost. They want to reuse Auto Loader's file tracking so no file is processed twice across runs. Which approach meets these requirements?

A data engineer builds an Auto Loader stream that ingests JSON files from a cloud storage path into a Bronze Delta table. The pipeline runs every hour using Trigger.AvailableNow. After a job restart, the engineer notices that previously ingested files are NOT being reprocessed, even though the source directory still contains all historical files. Which mechanism is responsible for ensuring each file is processed exactly once across restarts?

A data engineer is configuring an Auto Loader stream to incrementally ingest newly arriving Parquet files from an S3 landing zone into a Delta table. They write the following incomplete code: spark.readStream.format("cloudFiles").option(???).load("s3://landing/events/"). Which option is REQUIRED for Auto Loader to correctly read the incoming files?

A data engineer configures Auto Loader to ingest JSON files from cloud storage using schema inference. After the first run, they notice that numeric fields like 'order_total' and 'quantity' are all being inferred as STRING type in the target table, even though the JSON values are clearly numbers. The engineer wants Auto Loader to infer proper data types such as INTEGER and DOUBLE instead of treating everything as strings. Which option should they set to achieve this?

A data engineer configures an Auto Loader stream to ingest millions of new files that arrive daily into a large cloud storage bucket. The initial deployment uses the default directory listing mode, but as the number of files grows, the team notices that each micro-batch spends a long time simply discovering new files, increasing latency and cloud API costs. Which change should the engineer make to improve file-discovery efficiency at this scale?

A data engineering team receives daily drops of thousands of small JSON files into a cloud storage directory. They need a solution that automatically discovers and ingests only new files on each scheduled run, without manually tracking which files have already been loaded, and that requires minimal SQL or Python code for a Delta target. Given these requirements, which approach best fits a repeatable, incremental ingestion pipeline that scales as file counts grow over time?

A data engineer is configuring an Auto Loader stream to ingest millions of small JSON files that land continuously in a cloud storage bucket. The team notices that the default file discovery approach requires repeatedly listing the entire directory, which becomes slow and expensive as the number of files grows. Which Auto Loader configuration should the engineer use to improve file detection efficiency at scale?

A data engineer uses Auto Loader to ingest a very large backlog of JSON files that accumulated in a cloud storage location. When the stream starts, the cluster becomes overwhelmed because it tries to process an enormous number of files in a single micro-batch, causing memory pressure and slow performance. The engineer wants to limit how many files each micro-batch processes so the load is more manageable. Which option should be configured to control this behavior?

A data engineer configures Auto Loader to ingest JSON files from cloud storage into a Bronze Delta table. Occasionally, upstream systems send records where a field has a data type that conflicts with the inferred schema (e.g., a numeric field arriving as a non-numeric string). The engineer wants to ensure these mismatched values are not silently dropped and can be inspected later. Which Auto Loader feature captures data that does not match the expected schema?

A data engineer uses Auto Loader to ingest JSON files into a bronze Delta table. The stream is configured with a schema location, and the default schema evolution mode is in effect. One day the upstream team adds a new field to the JSON files. What happens to the Auto Loader stream when it encounters the new column?

A data engineer uses Auto Loader to incrementally ingest JSON files from a cloud storage location into a Bronze Delta table. The upstream system occasionally adds new fields to the JSON files. The engineer wants the pipeline to automatically detect and add these new columns to the target table without the stream failing, and to reprocess the affected data. Which configuration should the engineer use?

A data engineer is using Auto Loader to ingest JSON files into a bronze Delta table. Auto Loader correctly infers most columns, but the 'transaction_amount' field is being inferred as a STRING because some early files contained values wrapped in quotes. The engineer wants Auto Loader to treat this specific column as a DECIMAL while still inferring all other columns automatically. What is the most appropriate approach?

A data engineer is setting up an Auto Loader stream to ingest JSON files that land continuously in an S3 bucket. The team wants Auto Loader to automatically detect the columns and their data types on the first run without manually specifying a schema, and to persist the inferred schema so subsequent runs remain consistent. Which configuration is required to enable this behavior?

A data engineer configures an Auto Loader stream to ingest JSON files landing in a cloud storage directory. They want Auto Loader to infer the schema on the first run and then reuse that same inferred schema (tracking evolution over time) across subsequent job restarts, rather than re-inferring from scratch each time. Which option must be set to enable this behavior?

A data engineering team receives new JSON files in a cloud storage location every few minutes, around 50,000 small files per day. They need a low-latency ingestion pipeline that automatically detects and processes newly arriving files without reprocessing old ones, and they want to avoid the overhead of listing the entire directory on each run. Which ingestion approach best meets these requirements?

A data engineer needs to load a set of CSV files from a cloud storage location into an existing Delta table named sales_bronze. The files have a header row, and the engineer wants COPY INTO to infer column types automatically rather than treating everything as strings. Which addition to the COPY INTO command correctly enables header handling and automatic type inference for the CSV format?

A data engineer runs COPY INTO to load a batch of CSV files from a cloud storage location into a newly created empty Delta table. The command completes successfully. Later that day, without any new files being added to the source location, the engineer runs the exact same COPY INTO command again. What happens on the second execution?

A data engineer needs to load only the CSV files from a cloud storage directory that also contains JSON and Parquet files mixed together. The engineer wants to use COPY INTO to load these files into an existing Delta table, ensuring only files matching a specific naming pattern (e.g., files starting with 'sales_') are ingested. Which combination of COPY INTO options should be used?

A data engineer runs a nightly job that uses COPY INTO to load newly arrived CSV files from a cloud storage location into a Delta table. Due to a scheduling misconfiguration, the same job occasionally runs twice on the same night, pointing at the same source directory. The engineer notices that despite the duplicate runs, no duplicate rows appear in the target table. What is the primary reason COPY INTO avoids reloading the same data?

A data engineer uses COPY INTO to incrementally load daily CSV files from cloud storage into an existing Delta table. This week, the upstream team added two new columns to the source files. The engineer wants COPY INTO to automatically add these new columns to the target Delta table instead of failing or ignoring them. Which approach accomplishes this?