Question

A nightly Databricks job that reads a large Delta table and performs several wide transformations fails intermittently. In the Spark UI, you notice a stage marked as 'Failed' where several tasks show status 'FAILED' with an error 'ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits'. What is the most likely root cause you should investigate first?

Accepted Answer

Executors are running out of memory, likely due to data skew or excessive shuffle/spill for the wide transformations

Answer

The job cluster is running on an all-purpose cluster instead of a job cluster, causing memory contention

Answer

The Delta table has too few files, so partition pruning cannot occur

Answer

The checkpoint location for the streaming query has become corrupted