Question

A data engineer needs to build a Silver table that flags each product price record where the price changed compared to the immediately previous record for the same product, ordered by event timestamp. Which PySpark approach correctly computes the previous price within each product group?

Accepted Answer

Use F.lag('price').over(Window.partitionBy('product_id').orderBy('event_ts')) and compare it to the current price column

Answer

Use F.lag('price') without a Window specification and compare it to the current price column

Answer

Use groupBy('product_id').agg(F.first('price')) and join the result back to compare prices

Answer

Use F.lead('price').over(Window.partitionBy('event_ts').orderBy('product_id')) and compare it to the current price column