Question

A data engineering team runs a daily AWS Glue Spark ETL job that loads customer records from multiple upstream systems into a curated S3 dataset. Due to overlapping source feeds, the same customer_id can appear multiple times per run, and only the record with the most recent last_updated timestamp should be kept. The team needs to implement deduplication within the Glue job with minimal custom code while ensuring the surviving row is always the latest version. Which approach best meets these requirements?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use a window function partitioned by customer_id ordered by last_updated descending, and keep only the first row per partition.

Answer

Apply the DynamicFrame dropDuplicates() transform with no arguments to remove exact duplicate rows across all columns.

Answer

Enable Glue job bookmarks so that previously processed records are automatically excluded from subsequent runs.

Answer

Use the FindMatches ML transform to identify fuzzy duplicate customer records and merge them automatically.