Question

A data engineering team stores raw clickstream logs in Amazon S3 as JSON files, ingesting several terabytes daily. Data scientists repeatedly query subsets of this data filtered by event_date and region to build training datasets, but their Amazon Athena queries are slow and expensive because each query scans the entire dataset. The team wants to redesign the storage layout to minimize both query latency and cost with the least ongoing operational effort. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue ETL job to convert the JSON files to Parquet and write the output partitioned by event_date and region in S3.

Answer

Compress the JSON files with gzip and keep them in a single S3 prefix so Athena reads less data per query.

Answer

Load all the clickstream data into an Amazon RDS PostgreSQL instance and have the data scientists query it with SQL.

Answer

Enable S3 Intelligent-Tiering on the bucket and increase the Athena query concurrency limit.