Question

A user-facing checkout service on GKE has an SLO of p99 latency under 400ms, but users intermittently report slow checkouts. Aggregate dashboards show p50 and p90 latency well within target, and CPU/memory utilization is normal. The service calls three downstream microservices synchronously. You need to identify which downstream call is responsible for the tail-latency spikes with minimal added instrumentation effort. What should you do first?

Accepted Answer

Use Cloud Trace to inspect trace spans for slow requests, filtering on high-latency traces to see which downstream span dominates the request duration.

Answer

Increase the sampling rate of Cloud Monitoring metrics for the checkout service to capture more p99 data points on the dashboard.

Answer

Enable Cloud Profiler on the checkout service to find CPU hotspots in the request-handling code path.

Answer

Create a log-based metric on the checkout service's access logs and alert when request duration exceeds 400ms.