Question

A data engineering team runs a daily AWS Glue Spark ETL job that writes a large analytical dataset to Amazon S3 in Parquet format. The data is queried heavily via Amazon Athena, and queries frequently filter and aggregate across many columns. The team wants to minimize storage costs while achieving the best query performance, and they are willing to accept slightly higher write-time CPU usage during the ETL job. Which compression configuration should they apply to the Parquet output?

Accepted Answer

Write the Parquet files using GZIP compression to maximize the compression ratio and reduce storage footprint the most

Answer

Write the Parquet files using Snappy compression because it is the only codec Athena can read for Parquet

Answer

Write the Parquet files uncompressed so Athena does not spend time decompressing data during scans

Answer

Convert the output to CSV with GZIP compression to reduce both storage and Athena scan costs