Question

A data engineer builds a Lakeflow Job with two notebook tasks. The first task, 'extract', computes the number of new records loaded and needs to pass this value to a second task, 'notify', which depends on it. What is the correct mechanism for the 'extract' task to publish this value and for the 'notify' task to read it?

Accepted Answer

Use dbutils.jobs.taskValues.set() in 'extract' and dbutils.jobs.taskValues.get() in 'notify' to reference the value across tasks

Answer

Write the value to a global temporary view in 'extract' and query it from 'notify'

Answer

Store the value in a Python global variable that both notebooks share automatically

Answer

Pass the value using spark.conf.set() in 'extract' and spark.conf.get() in 'notify'