Question

A data engineering team runs a nightly Glue ETL job that loads sales data into Amazon S3 in Parquet format. Business analysts report that a QuickSight dashboard occasionally shows wildly inflated revenue totals for certain regions. The team suspects that some upstream source files contain duplicate transaction records and malformed numeric fields that only appear on certain days. They need a way to automatically detect and quantify these anomalies within the pipeline BEFORE the bad data reaches the dashboard, and to produce metrics they can trend over time. Which approach best meets this requirement with the least custom code?

Accepted Answer

Add AWS Glue Data Quality rules to the ETL job to evaluate completeness, uniqueness, and column value ranges, and publish the resulting quality scores to CloudWatch for trending.

Answer

Enable QuickSight ML Insights anomaly detection on the dashboard so analysts are alerted when revenue figures deviate from expected patterns.

Answer

Schedule an Athena CREATE TABLE AS SELECT query after each load to recompute totals and compare them against the previous day's results manually.

Answer

Increase the Glue job's DPU allocation and enable job bookmarks so that duplicate records are automatically skipped on subsequent runs.