Question

A data engineer runs a Spark notebook in Fabric that joins a large fact DataFrame with several transformations. The job completes but takes 45 minutes, far longer than expected. In the Spark UI for the run, one stage shows a very high 'Shuffle Spill (Disk)' value and several tasks running much longer than others. What is the MOST likely root cause to investigate first?

Accepted Answer

Data skew causing uneven partition sizes and memory pressure that spills shuffle data to disk

Answer

The Delta table has too many V-Order optimized files

Answer

The notebook is using the wrong sensitivity label, throttling reads

Answer

High-concurrency mode is disabled on the Spark session