Question

A media company ingests a continuous stream of application log events, currently reaching several thousand records per second. The analytics team needs the raw events delivered to an Amazon S3 data lake in near-real time, converted to Apache Parquet format, and partitioned by event date so that Amazon Athena can query them efficiently. The team wants to minimize the amount of custom code and infrastructure they have to manage. Which solution best meets these requirements?

Accepted Answer

Use Amazon Kinesis Data Firehose with record format conversion to Parquet enabled and dynamic partitioning to write the events to S3.

Answer

Use Amazon Kinesis Data Streams with a consumer running on an Auto Scaling group of EC2 instances that converts records to Parquet and writes them to S3.

Answer

Ingest the events into an Amazon MSK cluster and run a self-managed Kafka Connect S3 sink connector to write Parquet files.

Answer

Buffer events in an Amazon SQS queue and trigger an AWS Lambda function that converts each message to Parquet and stores it in S3.