Question

A data engineering team runs a nightly AWS Glue ETL job that loads customer transaction data into an S3-based data lake. Recently, downstream analysts reported that some batches contained duplicate transaction IDs and null values in the required 'amount' column. The team wants to automatically validate incoming data against defined rules (such as uniqueness and completeness) and halt the pipeline before publishing bad data, while capturing quality scores in CloudWatch for monitoring. Which approach best meets these requirements with the least custom code?

Accepted Answer

Add an AWS Glue Data Quality ruleset using DQDL within the Glue job, configure it to evaluate rules and fail the job when the quality score falls below the threshold, and publish results to CloudWatch.

Answer

Write a custom PySpark validation function inside the Glue job that counts nulls and duplicates, raises an exception on failure, and manually pushes metrics to CloudWatch using the boto3 put_metric_data API.

Answer

Enable an Amazon Athena scheduled query after the load completes to count nulls and duplicates, then use an EventBridge rule to alert the team if results exceed thresholds.

Answer

Configure an AWS Glue crawler to run before the job and reject files that do not match the expected schema, then rely on the schema check to prevent duplicates and nulls.