Question

A media company ingests approximately 50,000 events per second into an Amazon Kinesis Data Streams stream. A fleet of Amazon EC2 consumers running the Kinesis Client Library reads the stream, batches records into JSON files, and writes them to Amazon S3 every 5 minutes. A separate hourly AWS Glue job then compacts the small JSON files, converts them to Parquet, and partitions the data for Amazon Athena queries. The operations team reports that the EC2 consumer fleet frequently falls behind during traffic spikes, the Glue compaction jobs are expensive, and the pipeline requires significant maintenance. The team wants to reduce operational overhead and cost while improving reliability, without changing the source producers. Which approach best meets these goals?

Accepted Answer

Replace the EC2 KCL consumer fleet and Glue compaction job with Amazon Data Firehose configured to read from the Kinesis Data Stream, perform record-format conversion to Parquet, and buffer and deliver partitioned data directly to Amazon S3.

Answer

Keep the EC2 consumer fleet but move it to an Auto Scaling group with scaling based on the Kinesis GetRecords.IteratorAgeMilliseconds metric, and increase the Glue job frequency to every 15 minutes.

Answer

Migrate the Kinesis Data Stream to an Amazon SQS FIFO queue consumed by AWS Lambda that writes Parquet files to S3, and schedule the Glue compaction job to run every 10 minutes.

Answer

Replace the EC2 consumer fleet with an Amazon Managed Service for Apache Flink application that writes JSON to S3, and retain the hourly Glue compaction job to convert files to Parquet.