Working with Lakeflow Jobs

A data engineer is building a Lakeflow Declarative Pipeline (formerly Delta Live Tables) for a silver customer table. Business rules require that any record with a NULL email be excluded from the resulting table, but the pipeline must continue processing all other valid records without failing. Which expectation clause should the engineer use?

A data engineer is building a Lakeflow Declarative Pipeline (formerly Delta Live Tables) for a regulated financial dataset. The compliance team requires that if any record violates the constraint that transaction_amount must be greater than zero, the entire pipeline update must immediately stop and fail so that no partial data is published. Which expectation configuration should the engineer use?

A data engineer defines a Lakeflow Declarative Pipeline (Delta Live Tables) table with two expectations on a single dataset: one uses @dlt.expect_or_drop for a 'valid_amount' rule and another uses @dlt.expect (warn only) for a 'complete_customer' rule. During an update, a record fails the 'valid_amount' rule but passes 'complete_customer', while a second record passes 'valid_amount' but fails 'complete_customer'. What happens to these two records in the target table?

A data engineer defines a Lakeflow Declarative Pipeline (formerly Delta Live Tables) table with the following expectation: @dlt.expect("valid_amount", "amount > 0"). During a pipeline update, several records arrive with amount <= 0. What happens to those invalid records, and where can the engineer observe the violation?

A data engineer is building a Lakeflow Declarative Pipeline (formerly Delta Live Tables) for the silver layer. The source is an append-only bronze table that continuously receives new order records, and the team wants the silver dataset to process only newly arrived rows incrementally rather than recomputing the entire dataset on every pipeline update. Which dataset type should the engineer define?

A data engineer is iterating on a Lakeflow Declarative Pipeline (formerly Delta Live Tables) and wants fast feedback while testing changes. They notice that when they run the pipeline in one mode, the compute cluster stays running between updates and no automatic retries occur on failure, but when switched to another mode the cluster terminates after each update and tasks are retried automatically. Which mode keeps the cluster running with no retries, and what is its intended purpose?

A data engineer is building a Lakeflow Job with five sequential tasks, each running a short notebook. Currently every task spins up its own new job cluster, and the engineer notices most of the job's total runtime is spent waiting for clusters to start. Which configuration change will most effectively reduce this startup overhead while keeping compute isolated to this job?

A data engineering team runs a Lakeflow Job that must process incoming events with the lowest possible end-to-end latency, restarting the workload immediately after each run finishes so there is effectively no idle gap between executions. The job currently uses a cron schedule of every 5 minutes, but stakeholders complain that data can be up to 5 minutes stale. Which trigger configuration should the team use for the job?

A data engineer is building a Lakeflow Job with four tasks. Task A ingests raw data. Tasks B and C both perform independent transformations that each depend only on Task A's output. Task D aggregates the results and must only run after both B and C have completed successfully. How should the engineer configure the task dependencies to achieve this orchestration?

A data engineering team receives partner files at unpredictable times throughout the day in a specific cloud storage location. They want their Lakeflow Job to run automatically as soon as new files land, rather than waiting for a fixed schedule or paying for a continuously running cluster. Which trigger type should they configure for the job?

A data engineer schedules a Lakeflow Job to run every 15 minutes to ingest and process incoming files. Occasionally a run takes longer than 15 minutes to complete, and the engineer notices that when this happens, a new run starts before the previous one finishes, causing duplicate processing and resource contention. The engineer wants to ensure that only one run of this job executes at a time, with any triggered run waiting until the current one completes. Which job configuration change accomplishes this?

A data engineering team runs a nightly Lakeflow Job with five sequential tasks. The team wants to be automatically alerted by email whenever the job as a whole fails, so an on-call engineer can investigate before the next business day. They do not want an alert every time an individual task retries. Which configuration best meets this requirement?

A data engineer has a Lakeflow Job configured with a job-level parameter named 'process_date' whose default value is set to the current date. The job runs automatically each morning on a schedule. One morning, the scheduled run fails because of an upstream data delay. Later that day, the source data for the previous day becomes available, and the engineer needs to manually trigger the job to reprocess that specific earlier date without permanently changing the job's default configuration. What is the most appropriate way to accomplish this?

A data engineer builds a Lakeflow Job with five notebook tasks. Every task needs to read the same target catalog and schema names, which change between the dev and prod environments. The engineer wants to define these values in ONE place so all tasks can reference them, and so the values can be overridden when triggering a run for a different environment. Which approach best meets this requirement?

A data engineer runs a Lakeflow Job with five sequential tasks. The job fails on the fourth task due to a downstream API outage that has since been resolved. The first three tasks completed successfully and were expensive to run. The engineer wants to complete the job without re-executing the three successful tasks. What is the most efficient way to accomplish this?

A data engineering team has a Lakeflow Job with a cron schedule that runs a nightly ETL pipeline. Due to an upstream source system undergoing a two-week maintenance window, the team wants to temporarily stop the scheduled runs but keep the job definition, task configuration, and schedule intact so they can resume it later without recreating anything. What is the recommended action?

A data engineer has a Lakeflow Job that must run automatically every morning at 6:00 AM to refresh reporting tables. Occasionally, an analyst also needs to run the same job on demand during the day to reflect corrected source data. Which combination correctly satisfies both needs without creating a second job?

A data engineer configures a Lakeflow Job to run every day at 02:00 using a scheduled trigger. One morning the underlying job cluster fails to start due to a temporary cloud capacity issue, and the 02:00 run never executes. The team wants the pipeline to automatically attempt the run again shortly after a failure instead of waiting until the next day. Which configuration change best achieves this goal?

A data engineer builds a Lakeflow Job with three tasks: an ingest task, a transform task, and a cleanup task. The transform task depends on the ingest task. The cleanup task should always run to remove temporary files, even if the transform task fails. How should the engineer configure the dependencies and run condition for the cleanup task?

A data engineer configures a Lakeflow Job with a single task that ingests data from an external API. The API occasionally returns transient HTTP 503 errors that succeed on a subsequent attempt. The engineer wants the task to automatically re-run up to 3 times, waiting 2 minutes between attempts, before the job is marked as failed. Which task configuration should the engineer use?