Question

A data engineer configures a Lakeflow Job to run every day at 02:00 using a scheduled trigger. One morning the underlying job cluster fails to start due to a temporary cloud capacity issue, and the 02:00 run never executes. The team wants the pipeline to automatically attempt the run again shortly after a failure instead of waiting until the next day. Which configuration change best achieves this goal?

Accepted Answer

Add a retry policy to the job's tasks so failed runs are automatically retried after a short interval

Answer

Change the schedule from a daily trigger to a continuous trigger so the job runs without pausing

Answer

Increase the cron schedule frequency to run every hour so the pipeline eventually succeeds

Answer

Enable file arrival triggers so the job starts as soon as new source data lands