Question

A data engineering team runs a daily AWS Glue ETL job that includes an AWS Glue Data Quality ruleset. Business stakeholders want an ongoing, automated way to observe whether the overall data quality of the pipeline is degrading over several weeks, without failing the job or blocking downstream processing. They want to visualize the trend and receive a notification only when quality drops below a defined threshold. Which approach best meets these requirements?

Accepted Answer

Configure the Glue Data Quality evaluation to publish result metrics to Amazon CloudWatch, then create a CloudWatch dashboard for the score trend and a CloudWatch alarm on the metric threshold that notifies an SNS topic.

Answer

Set the Glue Data Quality ruleset action to stop the job on failure, and rely on the job-failure notification to indicate quality degradation over time.

Answer

Write the Data Quality results to Amazon S3 and run a weekly Athena query manually to check whether the average score has dropped.

Answer

Enable Glue job bookmarks and inspect the CloudWatch job duration metric, treating longer runtimes as a proxy for declining data quality.