Question

A company runs a fleet of 200 EC2 instances behind an Application Load Balancer. When a shared backend dependency degrades, dozens of instances simultaneously breach a CloudWatch alarm for high 5xx rates. Each alarm state change triggers an EventBridge rule that invokes a Lambda function, which in turn starts a Systems Manager Automation runbook to collect diagnostics and restart the affected service. During the last incident, the flood of concurrent Automation executions exhausted the account's SSM Automation concurrency, and the diagnostics runbook itself failed to complete on many instances. The DevOps team wants the automated response to still gather diagnostics and remediate, but without launching a redundant execution per instance for what is fundamentally a single upstream incident. Which approach BEST addresses this?

Accepted Answer

Replace the per-instance CloudWatch alarms with a single composite alarm that aggregates the fleet's 5xx metric, and have the EventBridge rule invoke one Automation execution that uses a rate control targeting the instances by tag with a MaxConcurrency and MaxErrors setting.

Answer

Increase the account's SSM Automation concurrent execution quota via a Service Quotas request so all per-instance executions can run in parallel without throttling.

Answer

Add a Lambda reserved concurrency limit of 1 so only one diagnostics runbook can be started at a time, queuing the remaining alarm events.

Answer

Configure an EventBridge dead-letter queue on the rule so throttled Automation invocations are retried later once concurrency frees up.