Question

A payments API team notices intermittent user complaints about slow checkout, but their existing Cloud Monitoring dashboards showing average request latency remain flat and green. The service handles 50,000 requests per minute. The SRE lead wants to configure observability so that the team can both detect the slowness users experience and diagnose which downstream dependency causes it. Which combination of observability configurations best addresses this need?

Accepted Answer

Replace the average latency metric with a 99th percentile latency distribution metric on the dashboard, and enable Cloud Trace with span attributes on downstream calls to identify the slow dependency.

Answer

Increase the alerting threshold on the average latency metric and add more notification channels so more engineers are paged when latency rises.

Answer

Enable Cloud Profiler on the payments service and rely on CPU flame graphs to determine why individual requests are slow.

Answer

Create a log-based counter metric that counts the number of requests per minute and alert when the request count drops below expected volume.