Question

A media company is building a new event-driven analytics platform. Multiple microservices emit JSON events at unpredictable rates (from a few hundred to several hundred thousand per minute). The data team needs to run ad-hoc SQL queries directly against the raw data within minutes of arrival, without managing servers or provisioning capacity. The event schema evolves frequently as developers add new fields, and the solution must remain cost-effective at idle. Which architecture best meets these requirements?

Accepted Answer

Ingest events through Amazon Kinesis Data Firehose that buffers and writes partitioned Parquet objects to Amazon S3, catalog the data with AWS Glue crawlers, and query it using Amazon Athena.

Answer

Send events to an Amazon SQS queue consumed by an Auto Scaling group of EC2 instances that batch-insert rows into an Amazon Redshift provisioned cluster for SQL analysis.

Answer

Stream events into Amazon Kinesis Data Streams and use a fixed fleet of Kinesis Client Library consumers to load records into an Amazon RDS for PostgreSQL instance for querying.

Answer

Write each event directly to Amazon DynamoDB from the microservices and run analytical SQL queries using PartiQL against the tables as data arrives.