Question

A data engineer notices that a nightly Spark job is running much slower than usual. In the Spark UI's 'Executors' tab, they see that several executors show 'Task Time (GC Time)' where the GC Time is nearly 40% of the total task time. What is the most likely conclusion from this observation?

Accepted Answer

The executors are spending excessive time on garbage collection due to memory pressure, indicating they are under-provisioned or holding too much data in memory

Answer

The job is bottlenecked on network shuffle and needs more shuffle partitions

Answer

The data is heavily skewed across a single partition causing a straggler task

Answer

The Delta table has too many small files that need to be compacted with OPTIMIZE