Question

A data engineer runs several AWS Glue Spark ETL jobs nightly. Occasionally a job succeeds but produces fewer output records than expected, and the team wants a way to retroactively investigate the root cause (skewed partitions, data movement, executor behavior) without re-running the job or adding custom logging to each script. What is the MOST operationally efficient way to capture this detailed diagnostic information?

Accepted Answer

Enable Glue job observability metrics (Amazon CloudWatch observability metrics) on the jobs so per-job performance and data metrics are automatically emitted to CloudWatch.

Answer

Add print statements throughout each Spark script and review the driver output in CloudWatch Logs after each run.

Answer

Enable AWS CloudTrail data events for the Glue jobs and analyze the API calls in Athena.

Answer

Increase the number of DPUs allocated to each job so Spark automatically records more detailed execution statistics.