Question

A data engineer runs a nightly AWS Glue Spark ETL job that has recently started failing after processing about 80% of its input. The CloudWatch job metrics show glue.driver.jvm.heap.usage climbing steadily toward 1.0 and glue.ExecutorAllocationManager.executors.numberMaxNeededExecutors well above the number of allocated executors, while glue.driver.aggregate.shuffleBytesWritten is very high. The job code performs a large groupBy followed by a collect() to build a lookup dictionary on the driver. What is the MOST likely root cause and appropriate fix?

Accepted Answer

The driver is running out of memory because collect() pulls the full grouped dataset onto the driver; refactor the logic to keep the aggregation distributed (e.g., use a join instead of collecting to the driver).

Answer

The job lacks enough DPUs for the crawler; enable a scheduled crawler to run before the job so partitions are cached.

Answer

Glue bookmarks are disabled, causing reprocessing of all data; enable job bookmarks to reduce the driver heap pressure.

Answer

The S3 source uses too many small files; convert the source to a single large CSV file to reduce shuffle.