Question

A data engineer needs to monitor a Lakeflow Declarative Pipeline (formerly Delta Live Tables) to report on how many records failed each expectation over the last several runs. Rather than manually inspecting the pipeline UI each time, they want to query this information programmatically to build an automated data-quality dashboard. What is the most appropriate source for these metrics?

Accepted Answer

Query the pipeline's event log, which records expectation results and data-quality metrics as structured events

Answer

Enable the Spark UI and scrape the SQL tab for expectation pass/fail counts after each run

Answer

Run DESCRIBE HISTORY on each target table to read the number of dropped rows per expectation

Answer

Parse the driver's stdout log files stored in the cluster log delivery location for expectation warnings