Question

A data engineering team ingests high-volume clickstream events as JSON through Amazon Kinesis Data Firehose delivering to Amazon S3. Analysts query the data with Amazon Athena and complain about slow queries and high scan costs. The team wants Firehose to reduce storage size and improve Athena query performance with minimal custom code. What should they configure?

Accepted Answer

Enable Firehose record format conversion to Apache Parquet using an AWS Glue Data Catalog table as the schema source, and enable dynamic partitioning by event date.

Answer

Attach a Lambda transformation to Firehose that compresses each JSON record with GZIP before delivery to S3.

Answer

Increase the Firehose buffer size to 128 MB and buffer interval to 900 seconds so fewer, larger JSON files are written to S3.

Answer

Deliver the raw JSON to S3 and schedule a nightly AWS Glue ETL job to rewrite the data into Avro format partitioned by hour.