Question

A financial services company runs nightly batch reconciliation using AWS Batch. When a Batch job fails, the operations team learns about it only the next morning from downstream reports, causing missed SLAs. The team wants an automated response that (1) captures every job that reaches the FAILED state, (2) automatically resubmits the job once if the failure reason indicates a transient host error, and (3) if the retry also fails or the failure is non-transient, opens an incident by publishing to an SNS topic that pages on-call. The solution must require no polling and add minimal custom infrastructure. What is the MOST appropriate design?

Accepted Answer

Create an EventBridge rule matching the AWS Batch Job State Change event where status is FAILED, targeting a Lambda function that inspects the failure reason, resubmits the job once for transient host errors, and publishes to the SNS topic for non-transient failures or exhausted retries.

Answer

Configure a CloudWatch alarm on the Batch FailedJobs metric that triggers an SSM Automation runbook to resubmit all failed jobs and notify SNS whenever the metric exceeds zero.

Answer

Schedule a Lambda function every 5 minutes with EventBridge Scheduler to call ListJobs with a FAILED filter, resubmit qualifying jobs, and publish to SNS for the rest.

Answer

Enable Batch job retry strategy with maxAttempts set to 2 and add an EventBridge rule on the Job State Change event that always publishes directly to SNS on FAILED so on-call is paged for every failure.