Question

A data engineer is transforming a large fact table (2 billion rows, ~500 GB) in a Fabric notebook using PySpark. Each fact row must be enriched with descriptive attributes from a small product lookup DataFrame (~8,000 rows, 2 MB). During the join, the Spark job runs slowly and generates heavy shuffle activity across executors. The engineer wants to eliminate the shuffle for the lookup while keeping the transformation in PySpark. What should the engineer do?

Accepted Answer

Wrap the small product DataFrame in broadcast() so Spark distributes a copy to every executor and performs a broadcast hash join

Answer

Repartition both DataFrames on the join key using a large number of partitions before joining

Answer

Cache the large fact DataFrame with persist(StorageLevel.MEMORY_AND_DISK) before executing the join

Answer

Convert the join to a cross join and filter the matching product keys with a where clause