Question

A company runs a fleet of EC2-based web servers behind an Application Load Balancer. Operations wants automated remediation: whenever a specific CloudWatch alarm named 'HighUnhealthyHostCount' transitions to ALARM state, a Lambda function should immediately gather diagnostics (recent target group health, application logs) and post a summary to a Slack channel. The team wants the reaction to occur as soon as the alarm changes state, without polling, and they must be able to filter so that only transitions to the ALARM state (not OK or INSUFFICIENT_DATA) trigger the workflow. Which approach meets these requirements with the least custom code?

Accepted Answer

Create an EventBridge rule that matches the 'CloudWatch Alarm State Change' event source, with an event pattern filtering on the alarm name and detail.state.value equal to ALARM, and set the diagnostics Lambda function as the target.

Answer

Configure the CloudWatch alarm to send notifications to an SNS topic, subscribe the Lambda function to the topic, and have the Lambda parse the message body to check whether the new state is ALARM before running diagnostics.

Answer

Create a scheduled EventBridge rule that invokes the Lambda every minute to call DescribeAlarms and, when the state is ALARM, run diagnostics and post to Slack.

Answer

Enable CloudWatch alarm actions to trigger an EC2 recovery action, and add a Systems Manager Automation runbook that captures diagnostics and calls a Slack webhook after recovery completes.