Question

An SRE team runs a payments API with a well-defined availability SLO. Recently, on-call engineers are paged frequently for brief blips that recover within a minute and never threaten the error budget, causing alert fatigue. Meanwhile, a slow, sustained burn last quarter went unnoticed until users complained. The team wants to redesign paging so that engineers are only woken up when a problem meaningfully threatens the SLO, while still tracking slower degradations. What should they implement?

Accepted Answer

Configure multi-window, multi-burn-rate error budget alerts: a fast-burn condition (e.g., 2% budget in 1 hour) that pages, and a slow-burn condition (e.g., 10% budget in several hours) that files a ticket.

Answer

Lower the SLO target so fewer events count as violations, reducing the number of pages the on-call receives.

Answer

Add a single threshold alert on raw request error count that pages whenever more than 5 errors occur in any 60-second window.

Answer

Increase the alert duration window on the existing per-minute error-rate alert to 30 minutes so short blips are ignored entirely.