Question

A data engineering team runs a Lakeflow Job that must process incoming events with the lowest possible end-to-end latency, restarting the workload immediately after each run finishes so there is effectively no idle gap between executions. The job currently uses a cron schedule of every 5 minutes, but stakeholders complain that data can be up to 5 minutes stale. Which trigger configuration should the team use for the job?

Accepted Answer

Configure the job with a Continuous trigger so a new run starts as soon as the previous run completes.

Answer

Reduce the cron schedule interval to every 1 minute to minimize staleness.

Answer

Add a File arrival trigger that fires only when new files land in the monitored location.

Answer

Configure a Continuous trigger but also keep the 5-minute cron schedule as a backup.