Question

A data engineer manages a Lakeflow Declarative (DLT) pipeline that applies several data quality expectations across bronze and silver tables. After a production run, stakeholders report that some downstream metrics look off, and the engineer needs to determine how many records failed each expectation over the last several pipeline updates without manually re-running the pipeline. Which approach provides this historical data quality information most directly?

Accepted Answer

Query the pipeline's event log, filtering on the 'flow_progress' events to read the data quality metrics recorded for each expectation.

Answer

Open the Spark UI for the most recent run and inspect the Stages tab for records dropped during each transformation.

Answer

Run DESCRIBE HISTORY on each target Delta table to view the count of rows rejected by each expectation.

Answer

Re-run the pipeline in development mode and watch the notebook cell output for expectation violation counts.