Question

A data engineer runs a Lakeflow Job with five sequential tasks. The job fails on the fourth task due to a downstream API outage that has since been resolved. The first three tasks completed successfully and were expensive to run. The engineer wants to complete the job without re-executing the three successful tasks. What is the most efficient way to accomplish this?

Accepted Answer

Use the 'Repair run' feature on the failed job run, which re-executes only the failed and not-yet-run tasks while reusing the successful task results.

Answer

Trigger a completely new run of the job, which will automatically skip tasks that already succeeded in the previous run.

Answer

Manually edit the job to remove the first three tasks, run the remaining tasks, then add the tasks back.

Answer

Increase the retry policy on the fourth task and re-run the entire job from the beginning.