Question

A data engineering team runs a nightly Spark job on Amazon EMR that joins a 2 TB fact table of transactions with a 40 MB reference table containing store metadata. The job is slow because Spark performs a shuffle-based sort-merge join, redistributing the massive fact table across the cluster over the network. The reference table easily fits in each executor's memory. What is the most effective change to accelerate this join with minimal code effort?

Accepted Answer

Apply a broadcast hint to the 40 MB reference table so it is sent to all executors, enabling a broadcast hash join and avoiding shuffling the fact table

Answer

Increase the number of shuffle partitions with spark.sql.shuffle.partitions to spread the sort-merge join across more tasks

Answer

Repartition the fact table by the join key before the join to co-locate matching records

Answer

Convert both tables to bucketed tables on the join key to eliminate the shuffle