Question

A data engineer notices that a nightly ETL job has slowed considerably. In the Spark UI, they observe that a single stage takes 90% of the total job time. Within that stage's summary metrics, the 'Shuffle Read Size' is very large (hundreds of GB) and the stage was triggered by a large groupBy aggregation across many columns. Task durations are relatively uniform (no single straggler). What is the most likely cause of the slowdown revealed by these metrics?

Accepted Answer

A wide transformation is forcing a massive data shuffle across the cluster during the aggregation

Answer

Data skew is concentrating records into one partition, creating a straggler task

Answer

The executors are running out of memory and repeatedly restarting

Answer

Too many small input files are causing excessive file-listing overhead before the stage begins