Question

A retail company runs a microservices platform on GKE and has enabled Cloud Trace to diagnose latency. During a major sales event, engineers report that traces for many slow requests are missing, making it hard to find the bottleneck. The default automatic sampling is capturing only a small fraction of requests under high load. The team wants to increase the likelihood of capturing traces for the slowest requests without a massive increase in ingestion cost or code changes across all services. What should the architect recommend?

Accepted Answer

Configure custom sampling in the instrumentation (e.g., OpenTelemetry) to increase the sampling rate and apply tail-based sampling that prioritizes high-latency and error spans.

Answer

Disable Cloud Trace and rely on Cloud Logging structured logs with latency fields to reconstruct request timing manually.

Answer

Increase the number of GKE nodes so that each service has spare CPU to emit 100% of traces during the event.

Answer

Export all traces to BigQuery and query them there to find slow requests after the event ends.