Question

A company runs hundreds of on-premises web servers that continuously append records to local Apache log files. The data engineering team must stream these log records into Amazon Kinesis Data Streams for near real-time analytics. The solution should require minimal custom code, automatically monitor log files for new lines, handle file rotation, and retry on delivery failures. Which approach best meets these requirements?

Accepted Answer

Install the Amazon Kinesis Agent on each web server and configure it to tail the log files and forward records to the Kinesis data stream.

Answer

Write a custom application using the Kinesis Producer Library (KPL) that reads the log files and calls PutRecords, then deploy it to every server.

Answer

Configure Amazon Kinesis Data Firehose to poll the log files on each server and deliver them to the Kinesis data stream.

Answer

Use AWS DataSync to transfer the log files to Amazon S3 and trigger a Lambda function to publish each record to the Kinesis data stream.