Question

A data engineering team maintains a customer dimension table in an S3-based data lake. The business requires that whenever a customer's attributes (such as address or loyalty tier) change, the previous values must be preserved as historical records rather than overwritten, so analysts can query the state of a customer at any point in time. The team uses AWS Glue Spark ETL jobs to process incoming daily batches of customer updates. Which transformation approach should the team implement in the Glue job?

Accepted Answer

Implement a Slowly Changing Dimension (SCD) Type 2 pattern: compare incoming records to current records, expire the old version by setting an end-date and active flag, and insert the new version with a new effective-date.

Answer

Use dropDuplicates on the customer ID to keep only the most recent record for each customer in every batch.

Answer

Perform an in-place overwrite of matching customer records using an SCD Type 1 upsert so the table always reflects the latest values.

Answer

Append every incoming record to the table without any comparison, relying on the ingestion timestamp column alone.