Question

A data engineering team stores clickstream data in Amazon S3 using a Hive-style layout: s3://logs/year=2024/month=05/day=12/. New daily partitions are added continuously. Analysts query the data through Amazon Athena, but query planning has become slow because millions of partitions must be fetched from the AWS Glue Data Catalog, and running a crawler after every load adds operational overhead. The team wants to eliminate partition-management overhead while keeping fast query planning. What should they do?

Accepted Answer

Enable partition projection on the Athena table by defining the partition key types and ranges in the table properties.

Answer

Increase the Glue crawler schedule to run every hour so partitions are always current.

Answer

Convert the data to a single unpartitioned Parquet file to remove the need for partition metadata.

Answer

Migrate the table metadata from the Glue Data Catalog to a self-managed Hive metastore on Amazon EMR.