Question

A data engineer is building a Lakeflow Job with five sequential tasks, each running a short notebook. Currently every task spins up its own new job cluster, and the engineer notices most of the job's total runtime is spent waiting for clusters to start. Which configuration change will most effectively reduce this startup overhead while keeping compute isolated to this job?

Accepted Answer

Configure a shared job cluster in the job settings and assign all five tasks to use that same cluster

Answer

Switch all tasks to run on an existing all-purpose cluster that stays running continuously

Answer

Set the job's maximum concurrent runs to 5 so the tasks start their clusters in parallel

Answer

Add a retry policy with a short delay to each task so cluster startup is amortized