Question

A retail company's on-call engineers are experiencing severe burnout. They receive hundreds of alerts per week from Cloud Monitoring, most triggered by individual metric thresholds (e.g., CPU > 80%, single 500 error) that resolve themselves without customer impact. Leadership asks the architect to redesign the alerting process to reduce noise while ensuring engineers are still paged when customers are genuinely affected. Which approach best addresses this business and technical process problem?

Accepted Answer

Define Service Level Objectives with error budgets and configure alerts based on the burn rate of the error budget, so pages fire only when customer-facing reliability is at risk.

Answer

Raise all metric thresholds (e.g., CPU > 95%) so fewer alerts trigger, and route the remaining alerts to the same on-call rotation.

Answer

Keep all existing threshold alerts but send them to email instead of paging so engineers can review them during business hours.

Answer

Increase the on-call rotation size so each engineer handles fewer alerts, and add a second escalation tier for overflow pages.