Question

A data engineer runs a Glue Spark ETL job that processes a few large, unsplittable gzip files. The job takes several hours and often finishes with the maximum number of DPUs allocated, yet costs are high. In the Glue job metrics (CloudWatch), the engineer observes that only one or two executors show high CPU while glue.driver.aggregate.numCompletedStages stays low and most executors show near-zero activity for long stretches. What is the MOST likely cause and the best corrective action?

Accepted Answer

The input gzip files cannot be split, so parallelism is limited to a few tasks; recompress the source data into a splittable format (e.g., bzip2 or many smaller files/Parquet) so Spark can distribute work across executors.

Answer

The job has too few DPUs allocated; increase the maximum capacity so more executors can process the same files in parallel.

Answer

Job bookmarks are disabled, causing reprocessing of the entire dataset; enable bookmarks to reduce runtime and executor idle time.

Answer

The Glue Data Catalog partitions are not registered; run a crawler so Spark can read all partitions in parallel.