Question

A data engineering team uses Databricks Asset Bundles (DABs) to deploy the same job to a development workspace and a production workspace. In development, jobs should run on a small single-node cluster and write to a dev catalog; in production, jobs should run on a larger cluster and write to a prod catalog. The team wants a single bundle definition that adapts these settings based on where it is deployed. What is the recommended way to achieve this?

Accepted Answer

Define multiple 'targets' in databricks.yml, each overriding variables and resource settings, and deploy with 'databricks bundle deploy -t <target>'.

Answer

Maintain two separate databricks.yml files in different Git branches and merge them manually before each deployment.

Answer

Hardcode the production values in the bundle and manually edit the YAML each time before deploying to development.

Answer

Use a single target and rely on the Databricks workspace to automatically detect the environment and swap cluster sizes.