Question

A data engineer builds a Lakeflow Job with five notebook tasks. Every task needs to read the same target catalog and schema names, which change between the dev and prod environments. The engineer wants to define these values in ONE place so all tasks can reference them, and so the values can be overridden when triggering a run for a different environment. Which approach best meets this requirement?

Accepted Answer

Define job parameters at the job level and reference them in each task using parameter syntax such as {{job.parameters.catalog}}.

Answer

Hard-code the catalog and schema in each notebook and edit all five notebooks whenever the environment changes.

Answer

Set separate task parameters individually on each of the five tasks so each task carries its own copy of the values.

Answer

Use dbutils.jobs.taskValues to emit the catalog name from the first task so downstream tasks can read it.