Question

A data engineer ingests a daily CDC extract into a Fabric lakehouse Delta table. Each source row includes a business key (CustomerID) and a LastModified timestamp. Because of overlapping extract windows, the same CustomerID can appear multiple times in a single batch with different LastModified values. Before merging into the dimension table, the engineer must reduce each batch so that only the most recent record per CustomerID remains. Which PySpark approach produces the correct result?

Accepted Answer

Apply a Window partitioned by CustomerID ordered by LastModified descending, add row_number(), and filter to rows where the row number equals 1.

Answer

Call dropDuplicates(['CustomerID']) on the batch DataFrame to keep one row per CustomerID.

Answer

Group by CustomerID and apply max('LastModified'), then select the resulting aggregated DataFrame.

Answer

Sort the DataFrame by LastModified descending and call distinct() to remove repeated business keys.