Question

A data engineer runs a nightly PySpark notebook in Fabric that joins a 2 TB fact table with a 400 GB dimension table. The job completes, but Spark UI shows that most tasks finish in under a minute while a handful of tasks run for over 40 minutes, dominating total runtime. The delayed tasks all process the same small number of join keys that account for the majority of the fact rows. What is the most effective way to reduce the overall job duration?

Accepted Answer

Enable Adaptive Query Execution (AQE) with skew join optimization so Spark automatically splits the oversized skewed partitions into smaller balanced tasks

Answer

Increase the number of executors and executor cores in the Spark pool to add more parallelism across all tasks

Answer

Broadcast the 400 GB dimension table to every executor so the shuffle-based join is avoided entirely

Answer

Run OPTIMIZE with V-Order on both Delta tables to compact small files before the join executes