Question

A company runs a fleet of EC2 instances behind an Application Load Balancer. Occasionally an instance fails ALB health checks and is placed in an 'unhealthy' state, but by the time an engineer investigates, an Auto Scaling group has already replaced the instance and the diagnostic data is lost. The DevOps team needs an automated process that, the moment a target enters the 'unhealthy' state, captures the instance's application logs, memory usage, and running processes and stores them in an S3 bucket BEFORE the instance is terminated—without requiring any human interaction. Which solution meets these requirements with the least operational overhead?

Accepted Answer

Create an EventBridge rule that matches the 'ELB target group Unhealthy' event from Elastic Load Balancing, targeting an SSM Automation runbook that uses AWS-RunShellScript (or Run Command) via the SSM Agent to gather logs, memory, and process data from the affected instance and upload them to S3.

Answer

Create an EventBridge rule matching the ELB TargetGroup 'UnHealthyHostCount' CloudWatch alarm state change, targeting a Lambda function that SSHes into every instance in the target group to collect logs and upload them to S3.

Answer

Configure an Auto Scaling lifecycle hook on instance termination that runs a user-data script to copy logs to S3 during the Terminating:Wait state, and rely on the ASG to detect the unhealthy instance.

Answer

Enable ALB access logs to S3 and create a CloudWatch Logs metric filter on the 503 error count that triggers an SNS notification so an engineer can manually connect to the instance and collect diagnostics.