Question

A payments microservice runs on GKE and normally serves requests with p99 latency under 300ms. During brief, self-healing garbage-collection pauses, p99 latency spikes above 500ms for 20-40 seconds several times per hour. You created a Cloud Monitoring alerting policy on p99 latency with a threshold of 500ms, but the on-call team is being paged constantly for these transient spikes that recover on their own. You must reduce noisy pages without missing sustained latency degradation that actually harms users. What is the best change to the alerting policy?

Accepted Answer

Increase the condition's retest/duration window so the metric must stay above 500ms for at least 5 minutes before the condition triggers.

Answer

Raise the threshold from 500ms to 2000ms so only extreme spikes page the team.

Answer

Add a metric-absence condition to the policy so it only fires when data stops arriving.

Answer

Change the aggregation alignment period to 1 second to capture the spikes more precisely.