PySpark
PySpark is the Python API for Apache Spark, used to build data transformations programmatically. The Data Engineer exam covers PySpark alongside Spark SQL for transformation and modeling.
All Databricks Data Engineer Associate Terms
Lakehouse
A lakehouse is an architecture that combines the low-cost storage of a data lake with the management and performance features of a data warehouse.
Delta Lake
Delta Lake is an open storage layer that brings ACID transactions, time travel, and schema enforcement to data lakes.
Medallion architecture
The medallion architecture is a data design pattern that refines data through bronze, silver, and gold layers.
Auto Loader
Auto Loader is a Databricks feature that incrementally and efficiently ingests new files from cloud storage as they arrive.
Structured Streaming
Structured Streaming is the Spark engine for scalable, fault-tolerant stream processing using the same DataFrame API as batch.
MERGE
MERGE is a SQL operation that performs upserts — inserting, updating, or deleting rows in a Delta table based on a match condition.
Spark SQL
Spark SQL is the Spark module for querying structured data with SQL and the DataFrame API.
PySpark
PySpark is the Python API for Apache Spark, used to build data transformations programmatically.
Lakeflow Jobs
Lakeflow Jobs (formerly Databricks Workflows) is the orchestration service for scheduling and running multi-task data pipelines.
Lakeflow Declarative Pipelines
Lakeflow Declarative Pipelines (formerly Delta Live Tables) is a framework for building reliable declarative data pipelines with built-in quality expectations.
Databricks Asset Bundles
Databricks Asset Bundles are a way to package code, jobs, and configuration as version-controlled projects for deployment across environments.
Z-ordering
Z-ordering is a Delta Lake optimization that co-locates related data in the same files to speed up queries that filter on those columns.
Unity Catalog
Unity Catalog is the Databricks governance layer providing a three-level namespace, access control, and data lineage across the lakehouse.
Spark UI
The Spark UI is the web interface that shows jobs, stages, and tasks for diagnosing and tuning Spark workloads.