Question

A company wants to centralize application logs from 40 Lambda functions and 15 EC2-based services into a single Amazon S3 bucket for later analysis with Athena. All logs are already written to Amazon CloudWatch Logs. The data engineering team needs a low-maintenance, near-real-time solution that requires no custom polling code and delivers the logs in a compressed, queryable format to S3. Which approach meets these requirements with the LEAST operational overhead?

Accepted Answer

Create a CloudWatch Logs subscription filter on each log group that streams records to an Amazon Kinesis Data Firehose delivery stream, and configure Firehose to buffer, GZIP-compress, and deliver the data to the S3 bucket.

Answer

Schedule an AWS Lambda function every 5 minutes to call the CloudWatch Logs GetLogEvents API for each log group and write the results to S3 as JSON files.

Answer

Export each CloudWatch log group to S3 daily using the CreateExportTask API triggered by an EventBridge scheduled rule.

Answer

Install the Kinesis Agent on each EC2 instance and reconfigure all Lambda functions to write logs directly to an Amazon Kinesis Data Stream, then use a consumer application to write to S3.