Question

A data engineer notices that a Spark job processing a large join runs slowly. In the Spark UI, they open the stage details and see that 199 of 200 tasks completed in under 10 seconds, but one task ran for 12 minutes and shuffled far more data than the others. What is the most likely cause of this behavior?

Accepted Answer

Data skew, where one join key holds a disproportionately large number of records

Answer

The cluster has too few worker nodes to run all 200 tasks in parallel

Answer

The Delta table needs to be vacuumed to remove stale data files

Answer

Auto Loader checkpointing is slowing down the final task