Hyperopt is a hyperparameter-tuning library historically used on Databricks with SparkTrials to parallelize search across a cluster. It is now legacy — unmaintained after Databricks Runtime ML 16.4 LTS — with Optuna and Ray Tune the current recommended libraries.
The Databricks Data Intelligence Platform is the unified lakehouse platform combining data engineering, analytics, and machine learning on Apache Spark and Delta Lake.
Databricks Runtime for Machine Learning is a cluster runtime that preinstalls common ML libraries such as scikit-learn, TensorFlow, PyTorch, XGBoost, and MLflow.
MLflow is the open-source platform, built into Databricks, for managing the ML lifecycle — tracking, projects, models, and the model registry.
MLflow Tracking is the component that logs parameters, metrics, artifacts, and models for each run so experiments are reproducible and comparable.
An Experiment is the MLflow container that groups related runs, each capturing the parameters and metrics of one training attempt.
The Model Registry is where MLflow models are versioned, aliased, and governed — on Databricks this is Models in Unity Catalog, which supersedes the legacy Workspace Model Registry.
Unity Catalog is the Databricks governance layer for data and AI assets, providing centralized access control and lineage across workspaces.
AutoML is the Databricks feature that automatically trains and tunes candidate models for classification, regression, or forecasting and generates editable source notebooks.
The Feature Store is the Databricks capability for creating, sharing, and serving curated feature tables (governed in Unity Catalog) for training and inference.
Spark ML (spark.
A Pipeline in Spark ML chains transformers and an estimator into a single reusable workflow that can be fit and applied consistently.
Hyperopt is a hyperparameter-tuning library historically used on Databricks with SparkTrials to parallelize search across a cluster.
A pandas UDF (and applyInPandas) is a vectorized user-defined function that applies pandas logic across Spark partitions for scalable feature engineering or inference.
Batch Inference is scoring a large dataset at once by loading a registered MLflow model, often as a Spark UDF, and writing predictions back to a table.
Model Serving is the Databricks feature that exposes a registered model behind a REST endpoint for low-latency, real-time inference.