Medium Databricks Data Engineer Associate practice questions

A data engineering team runs a nightly production ETL pipeline that is triggered automatically by the Databricks Jobs scheduler. During code review, a teammate notices the pipeline is configured to run on an all-purpose cluster that stays running 24/7 so analysts can also attach notebooks to it interactively during the day. Management wants to reduce compute cost for the scheduled pipeline without affecting the analysts' interactive work. What is the most appropriate change?

A data engineering team uses Databricks Asset Bundles to deploy jobs and notebooks. When they run 'databricks bundle deploy -t dev', the engineer wants to confirm where the bundle's files and resources are placed so multiple developers on the same team do not overwrite each other's deployments in the 'dev' target. By default, how does a Databricks Asset Bundle isolate a developer's deployment in the 'dev' target?

A data engineering team's Databricks Asset Bundle project has grown large, with dozens of job and pipeline resource definitions all placed directly inside the top-level databricks.yml file. The lead engineer wants to split these resource definitions into separate YAML files under a resources/ directory so each team member can maintain their own jobs without causing merge conflicts in one giant file. Which databricks.yml mechanism allows these separate resource files to be combined into the bundle configuration?

A data engineer has defined a Databricks Asset Bundle containing a job resource in databricks.yml. After committing changes to Git, they want to push the updated job definition and its associated files to the Databricks workspace for the 'dev' target, but they do NOT want to trigger execution of the job yet. Which Databricks CLI command accomplishes this?

A data engineering team wants every new Databricks Asset Bundle project to start with a consistent folder structure, a pre-configured databricks.yml, and standard job definitions that match company conventions. They want engineers to scaffold new projects quickly without copying files manually. Which Databricks CLI capability should they use to accomplish this?

A data engineering team uses Databricks Asset Bundles. Each developer deploys the same bundle to a shared workspace from their own feature branch. They report that when two developers deploy simultaneously, their jobs collide — one developer's deployment overwrites the other's job and schedules start firing unexpectedly during testing. They want each developer's deployment to be isolated (uniquely named per user) and to have schedules automatically paused. Which configuration in the target should they use?

A data engineering team uses Databricks Asset Bundles to deploy jobs. During development, jobs run under each developer's personal identity, which is fine for testing. However, when deploying to the production target, the team wants all jobs to run under a shared service principal rather than the identity of whoever last ran `databricks bundle deploy`. Which configuration approach in the bundle's databricks.yml achieves this for the production target?

A data engineering team uses Databricks Asset Bundles (DABs) to deploy the same job to a development workspace and a production workspace. In development, jobs should run on a small single-node cluster and write to a dev catalog; in production, jobs should run on a larger cluster and write to a prod catalog. The team wants a single bundle definition that adapts these settings based on where it is deployed. What is the recommended way to achieve this?

A data engineer has finished editing the databricks.yml configuration and several job definitions for a Databricks Asset Bundle. Before deploying to the development target, they want to confirm that the bundle configuration is syntactically correct and that all references resolve properly, without actually creating or updating any resources in the workspace. Which CLI command should they run?

A data engineering team maintains a Databricks Asset Bundle whose databricks.yml defines several jobs. Each job needs to reference the same notification email address and the same Unity Catalog catalog name in multiple places. The team wants to define these values in ONE location and reuse them throughout the bundle configuration, so a single edit updates every reference. Which Asset Bundle feature should they use?

A data engineering team ingests new JSON order files that land in a cloud storage location throughout the day. They want to run a scheduled job every hour that processes only files that have arrived since the last run, then shuts down the cluster to save cost. They want to reuse Auto Loader's file tracking so no file is processed twice across runs. Which approach meets these requirements?

A data engineer builds an Auto Loader stream that ingests JSON files from a cloud storage path into a Bronze Delta table. The pipeline runs every hour using Trigger.AvailableNow. After a job restart, the engineer notices that previously ingested files are NOT being reprocessed, even though the source directory still contains all historical files. Which mechanism is responsible for ensuring each file is processed exactly once across restarts?

A data engineer is configuring an Auto Loader stream to incrementally ingest newly arriving Parquet files from an S3 landing zone into a Delta table. They write the following incomplete code: spark.readStream.format("cloudFiles").option(???).load("s3://landing/events/"). Which option is REQUIRED for Auto Loader to correctly read the incoming files?

A data engineer configures Auto Loader to ingest JSON files from cloud storage using schema inference. After the first run, they notice that numeric fields like 'order_total' and 'quantity' are all being inferred as STRING type in the target table, even though the JSON values are clearly numbers. The engineer wants Auto Loader to infer proper data types such as INTEGER and DOUBLE instead of treating everything as strings. Which option should they set to achieve this?

A data engineer configures an Auto Loader stream to ingest millions of new files that arrive daily into a large cloud storage bucket. The initial deployment uses the default directory listing mode, but as the number of files grows, the team notices that each micro-batch spends a long time simply discovering new files, increasing latency and cloud API costs. Which change should the engineer make to improve file-discovery efficiency at this scale?

A data engineering team receives daily drops of thousands of small JSON files into a cloud storage directory. They need a solution that automatically discovers and ingests only new files on each scheduled run, without manually tracking which files have already been loaded, and that requires minimal SQL or Python code for a Delta target. Given these requirements, which approach best fits a repeatable, incremental ingestion pipeline that scales as file counts grow over time?

A data engineer is configuring an Auto Loader stream to ingest millions of small JSON files that land continuously in a cloud storage bucket. The team notices that the default file discovery approach requires repeatedly listing the entire directory, which becomes slow and expensive as the number of files grows. Which Auto Loader configuration should the engineer use to improve file detection efficiency at scale?

A data engineer uses Auto Loader to ingest a very large backlog of JSON files that accumulated in a cloud storage location. When the stream starts, the cluster becomes overwhelmed because it tries to process an enormous number of files in a single micro-batch, causing memory pressure and slow performance. The engineer wants to limit how many files each micro-batch processes so the load is more manageable. Which option should be configured to control this behavior?

A data engineer configures Auto Loader to ingest JSON files from cloud storage into a Bronze Delta table. Occasionally, upstream systems send records where a field has a data type that conflicts with the inferred schema (e.g., a numeric field arriving as a non-numeric string). The engineer wants to ensure these mismatched values are not silently dropped and can be inspected later. Which Auto Loader feature captures data that does not match the expected schema?

A data engineer uses Auto Loader to ingest JSON files into a bronze Delta table. The stream is configured with a schema location, and the default schema evolution mode is in effect. One day the upstream team adds a new field to the JSON files. What happens to the Auto Loader stream when it encounters the new column?

A data engineer uses Auto Loader to incrementally ingest JSON files from a cloud storage location into a Bronze Delta table. The upstream system occasionally adds new fields to the JSON files. The engineer wants the pipeline to automatically detect and add these new columns to the target table without the stream failing, and to reprocess the affected data. Which configuration should the engineer use?

A data engineer is using Auto Loader to ingest JSON files into a bronze Delta table. Auto Loader correctly infers most columns, but the 'transaction_amount' field is being inferred as a STRING because some early files contained values wrapped in quotes. The engineer wants Auto Loader to treat this specific column as a DECIMAL while still inferring all other columns automatically. What is the most appropriate approach?

A data engineer is setting up an Auto Loader stream to ingest JSON files that land continuously in an S3 bucket. The team wants Auto Loader to automatically detect the columns and their data types on the first run without manually specifying a schema, and to persist the inferred schema so subsequent runs remain consistent. Which configuration is required to enable this behavior?

A data engineer configures an Auto Loader stream to ingest JSON files landing in a cloud storage directory. They want Auto Loader to infer the schema on the first run and then reuse that same inferred schema (tracking evolution over time) across subsequent job restarts, rather than re-inferring from scratch each time. Which option must be set to enable this behavior?

A data engineering team receives new JSON files in a cloud storage location every few minutes, around 50,000 small files per day. They need a low-latency ingestion pipeline that automatically detects and processes newly arriving files without reprocessing old ones, and they want to avoid the overhead of listing the entire directory on each run. Which ingestion approach best meets these requirements?