Question

A DevOps team runs a customer-facing checkout API behind an ALB. They want an automated incident triage workflow: when the API's p99 latency alarm enters ALARM state, the system should automatically capture diagnostics (recent application logs, target group health, and a thread dump from the affected EC2 instances) and post the collected artifacts to a Slack channel used by the on-call engineer. The team wants the diagnostics gathering to run across all impacted instances in the Auto Scaling group without writing custom fleet-iteration code, and to only be triggered by the specific latency alarm. Which approach best meets these requirements?

Accepted Answer

Create an EventBridge rule matching the CloudWatch Alarm State Change event for that specific alarm ARN with state ALARM, targeting an SSM Automation runbook that uses aws:runCommand with a tag-based target to collect logs and thread dumps fleet-wide, then invokes a Lambda to post artifacts to Slack.

Answer

Configure the CloudWatch alarm to send directly to an SNS topic subscribed by a single Lambda function that SSHes into each instance sequentially to run diagnostic commands and uploads results to Slack.

Answer

Enable CloudWatch Application Insights on the ALB and rely on its automatic problem detection to open findings, then manually copy the findings into Slack when the on-call engineer is paged.

Answer

Create an EventBridge Scheduler schedule that runs every minute to poll the alarm state via the CloudWatch API and, when ALARM, triggers a Step Functions workflow that iterates over instance IDs with custom code to run diagnostics.