Question

A data engineering team recently onboarded a new upstream vendor feed that lands as CSV files in Amazon S3 and is processed by an AWS Glue ETL job. Because the vendor's schema and value ranges are not well documented, the team wants a fast way to establish an initial baseline set of data quality checks (such as completeness, uniqueness, and column value ranges) without hand-authoring every rule from scratch. Which approach requires the least manual effort to produce a starting DQDL ruleset?

Accepted Answer

Run AWS Glue Data Quality rule recommendations against the dataset in the Data Catalog to generate a suggested DQDL ruleset that the team can review and adjust.

Answer

Write a custom PySpark script in the Glue job that computes column statistics and manually translate the results into DQDL rules.

Answer

Enable Amazon CloudWatch anomaly detection on the Glue job's numFailedTasks metric to infer data quality thresholds.

Answer

Configure an AWS Glue crawler with schema inference and rely on the crawler to emit data quality rules for each column.