Question

A company runs an automated incident response workflow. When an Amazon CloudWatch alarm enters ALARM state for high CPU on a specific EC2 instance, an EventBridge rule triggers an AWS Systems Manager Automation runbook that reboots the instance and posts to a Slack channel. During a recent incident, operators noticed the instance was rebooted three times within one minute and received three duplicate Slack messages, extending the outage. Investigation shows the alarm briefly flapped between ALARM and OK, and EventBridge invoked the target multiple times. The team must ensure that repeated events for the same incident do not trigger redundant remediation actions. Which approach BEST addresses this requirement?

Accepted Answer

Add a Systems Manager Automation runbook step that checks whether a remediation is already in progress for the instance (for example using a DynamoDB conditional write as a distributed lock/idempotency token) and exits early if one exists.

Answer

Increase the CloudWatch alarm evaluation period and datapoints-to-alarm so the alarm fires less frequently, and enable EventBridge exactly-once delivery on the rule.

Answer

Configure an EventBridge dead-letter queue on the rule target so that duplicate invocations are routed to the DLQ instead of the Automation runbook.

Answer

Set the EventBridge rule to use a rate-based schedule expression so the target can be invoked at most once per minute regardless of how many alarm events arrive.