Question

A data engineer runs a PySpark notebook in Fabric that performs a wide aggregation on a 5 GB dataset. The job completes but is unexpectedly slow. In the Spark UI, they observe 200 shuffle tasks, most of which process only a few kilobytes of data and finish almost instantly, while the total job spends significant time in scheduling overhead. The Spark pool has a small node configuration. What is the most effective way to improve performance for this workload?

Accepted Answer

Reduce the value of spark.sql.shuffle.partitions to better match the small data volume and cluster size

Answer

Increase spark.sql.shuffle.partitions to 800 so more tasks run in parallel

Answer

Cache the source DataFrame before the aggregation to avoid recomputation

Answer

Repartition the DataFrame to 500 partitions before performing the aggregation