Question

A data engineering team runs an AWS Glue Spark ETL job that transforms 3 years of raw transaction logs into a curated dataset used to build ML training sets. Data scientists query this dataset in Amazon Athena, almost always filtering by a specific transaction_date range and by region. Currently the Glue job writes all output as a single set of Parquet files to one S3 prefix, and Athena queries scan the entire dataset, driving up cost and query time. Which change to the Glue job most effectively reduces the data scanned per query?

Accepted Answer

Configure the Glue job to write output partitioned by transaction_date and region using partitionKeys, so Athena can prune partitions when filtering.

Answer

Increase the number of Glue worker nodes (DPUs) so the job produces more, smaller Parquet files across the same single prefix.

Answer

Change the output format from Parquet to CSV so Athena can read rows sequentially and stop scanning earlier.

Answer

Enable Glue job bookmarks so only newly arrived records are processed on each run.