Question

A checkout microservice on GKE shows p95 latency of 2.1s, but the service's own CPU and memory metrics are healthy. The team has instrumented the service with OpenTelemetry exporting to Cloud Trace, and each request creates spans for its three downstream calls (inventory, payments, and a Cloud SQL database). You need to determine which specific downstream dependency is responsible for the tail latency with the least additional instrumentation effort. What should you do?

Accepted Answer

Open Cloud Trace, filter for high-latency traces on the checkout service, and inspect the span waterfall to identify which child span consumes the most time.

Answer

Create a log-based metric that counts checkout requests exceeding 2 seconds and build a Cloud Monitoring dashboard grouped by HTTP status code.

Answer

Enable Cloud Profiler on the checkout service and analyze the wall-clock flame graph to find the slowest function.

Answer

Increase the Cloud Trace sampling rate to 100% and wait for the aggregate latency chart in Cloud Monitoring to reveal the bottleneck.