Question

A retail analytics team streams clickstream events as JSON records into Amazon Kinesis Data Firehose, which delivers them to an S3 data lake. Each record contains a 'customer_region' field and an 'event_date' field. The team wants the data physically partitioned in S3 by region and date (e.g., s3://lake/region=us-east/date=2024-05-01/) so that downstream Athena queries can prune partitions efficiently. They want to avoid running a separate post-processing job. What is the most operationally efficient way to achieve this?

Accepted Answer

Enable Firehose dynamic partitioning and use jq expressions (or an inline Lambda) to extract partition keys from customer_region and event_date, then reference them in the S3 prefix.

Answer

Deliver all records to a single S3 prefix and schedule an hourly AWS Glue ETL job that re-reads and rewrites the data into the partitioned layout.

Answer

Configure a Kinesis Data Streams consumer application with the KCL that manually writes each record to the correct partition path in S3.

Answer

Use Firehose default timestamp-based prefixes and rely on Athena's MSCK REPAIR TABLE to infer region and date partitions at query time.