Question

A data engineering team runs an AWS Glue Spark ETL job that reads clickstream events and writes the results to Amazon S3 in Parquet format. Downstream Athena queries filter heavily by event_date and country. Currently the job writes all output into a single S3 prefix, causing Athena to scan the entire dataset on every query. The team wants the Glue job itself to physically organize the output so that Athena can prune data by these two columns without a separate reprocessing step. What is the MOST appropriate way to configure the Glue write operation?

Accepted Answer

Write the DynamicFrame using the getSink API with partitionKeys set to ["event_date", "country"] so Glue writes Hive-style partitioned prefixes in S3.

Answer

Enable Glue job bookmarks so that only new records are written on each run, which forces partition creation automatically.

Answer

Increase the number of Glue DPUs so the job produces more output files, improving Athena partition pruning.

Answer

Run a Glue crawler after the job completes and set the crawler to create partitions based on the event_date and country columns.