Question

A data engineering team runs a nightly ingestion pipeline. A custom application writes structured log lines to Amazon CloudWatch Logs, including a line formatted as "REJECTED_RECORDS count=<n>" whenever malformed records are skipped. The team wants to be automatically alerted whenever the number of rejected records exceeds 500 in any single run, without modifying the application or building a separate log-processing service. What is the most operationally efficient way to achieve this?

Accepted Answer

Create a CloudWatch Logs metric filter that extracts the count value from matching log events, publish it as a custom metric, and configure a CloudWatch alarm with a threshold of 500 that notifies an SNS topic.

Answer

Stream the CloudWatch log group to Amazon Athena and schedule a query every hour to count rejected records, sending results by email.

Answer

Enable CloudWatch Logs Insights and run a scheduled query that filters for REJECTED_RECORDS, then manually review the dashboard each morning.

Answer

Create a Lambda function subscribed to the log group that parses every event and writes the count to DynamoDB, then poll DynamoDB with a second Lambda to send alerts.