Question

A data engineer needs to add a column to a Spark DataFrame of sales transactions that shows, for each row, the total number of transactions made by that customer — while still keeping every individual transaction row in the output. Which approach produces the correct result without collapsing rows?

Accepted Answer

Apply a window function: count('*').over(Window.partitionBy('customer_id')) as a new column

Answer

Use groupBy('customer_id').count() and use the resulting DataFrame as the final output

Answer

Use groupBy('customer_id').agg(count('*')) then drop duplicate customer rows

Answer

Apply select(countDistinct('transaction_id')) over the entire DataFrame