Question

A payment service running on GKE writes structured JSON logs to Cloud Logging. During a partial outage, the on-call engineer needs to quickly find all ERROR-severity entries that contain a specific transaction ID field (jsonPayload.txnId) across the last 30 minutes, then aggregate how many distinct pods emitted them. The engineer wants the fastest interactive way to both filter and run aggregation without exporting data elsewhere. Which approach should they use?

Accepted Answer

Use the Logs Explorer with a query on severity and jsonPayload.txnId, then switch to Log Analytics and run a SQL query grouping by resource.labels.pod_name

Answer

Create a log-based metric filtered on the txnId and view a Cloud Monitoring dashboard chart broken down by pod

Answer

Create a log sink to BigQuery and run a scheduled query to count distinct pods after the data lands

Answer

Enable Cloud Trace sampling at 100% and inspect spans grouped by transaction ID in the Trace explorer