Question

A retail company ingests raw customer order data into Amazon S3 as CSV files. A data engineer must build an AWS Glue ETL job that cleanses the data before loading it into a curated zone. The requirements are: replace empty strings and the literal text 'N/A' in the customer_email column with null, drop any row where order_id is missing, and standardize the country column so that values like 'usa', 'USA', and 'Usa' all become 'US'. Which approach implements all three cleansing requirements most efficiently within a single Glue Spark ETL job?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use withColumn with when/otherwise and regexp/upper functions to normalize country and null-out email values, and use dropna(subset=['order_id']) before writing back out.

Answer

Use the Glue DynamicFrame ResolveChoice transform to cast all columns to strings, then rely on the Glue Data Catalog crawler to automatically remove null order_id rows during the next crawl.

Answer

Write the raw data to S3 unchanged and create an Athena view with CASE expressions so the transformations are applied only at query time.

Answer

Enable a Glue job bookmark and configure the job's transformation_ctx so that Glue automatically standardizes casing and removes null-key records on incremental runs.