Question

A data engineer builds a Lakeflow Job with three tasks: an ingest task, a transform task, and a cleanup task. The transform task depends on the ingest task. The cleanup task should always run to remove temporary files, even if the transform task fails. How should the engineer configure the dependencies and run condition for the cleanup task?

Accepted Answer

Make cleanup depend on transform and set its 'Run if dependencies' condition to 'All done'

Answer

Make cleanup depend on transform and leave the default 'All succeeded' run condition

Answer

Remove all dependencies on cleanup so it runs first, before ingest and transform

Answer

Add a retry policy to the cleanup task so it retries until the transform succeeds