Question

Your SRE team runs a microservices platform on GKE. During a recent regional outage, on-call engineers received hundreds of separate alert notifications within minutes — one per affected pod and endpoint — overwhelming the incident channel and delaying triage. The underlying alerting policies are correctly configured to detect real problems. You need to reduce notification noise during correlated incidents WITHOUT suppressing detection of distinct issues. What should you do in Cloud Monitoring?

Accepted Answer

Configure the alerting policy's notification behavior to group incidents by relevant labels (e.g., cluster or service) so correlated conditions consolidate into fewer notifications.

Answer

Increase the alerting policy's duration (retest) window to several hours so transient spikes do not trigger notifications.

Answer

Delete the per-pod alerting policies and rely solely on a single uptime check for the whole platform.

Answer

Lower the alert threshold sensitivity so fewer resources cross the alerting boundary during an outage.