Question

A data engineer notices that a Databricks job has been running far longer than usual. Opening the Spark UI, they see one stage where 199 of 200 tasks completed quickly, but a single remaining task has been running for over 40 minutes with a very high 'Shuffle Read Size' compared to the other tasks. What is the most likely cause of this behavior?

Accepted Answer

Data skew is causing one partition to receive a disproportionately large share of the shuffled data

Answer

The cluster has too few executor cores to run all 200 tasks in parallel

Answer

The Delta table needs OPTIMIZE because it has too many small files

Answer

Photon is disabled, which slows down all aggregation tasks equally