Implementing observability practices and troubleshooting issues

Your team runs a microservices application on GKE with structured JSON logs flowing into Cloud Logging. During a partial outage, on-call engineers need to correlate error logs from the payment service with slow upstream requests across a 6-hour window, aggregating by HTTP status code and computing p95 latency per service. Ad-hoc filtering in the Logs Explorer is too slow and cannot perform these aggregations. What is the most appropriate approach to enable this analysis with minimal additional infrastructure?

A user-facing API running on GKE intermittently returns HTTP 500 errors. Your team has structured JSON logs in Cloud Logging that include a 'trace' field populated with the X-Cloud-Trace-Context ID, and distributed tracing is enabled with Cloud Trace. During an incident, you find a specific log entry showing a 500 error and want to immediately see the full end-to-end request path—including the downstream service calls and their latencies—that produced this error. What is the most efficient way to accomplish this?

Your compliance team requires that all audit logs from a production project be retained for 400 days and be queryable with SQL for ad-hoc investigations, while keeping operational costs low. Application debug logs should only be kept for 14 days. You want to configure Cloud Logging with the least operational overhead. What should you do?

Your team runs a high-traffic web service on GKE. Cloud Logging costs have risen sharply, and analysis shows that the majority of ingested volume comes from verbose HTTP access logs with severity INFO from the ingress load balancer. Compliance requires that all WARNING and higher logs remain searchable in Cloud Logging for 30 days, and a subset of the INFO access logs must still be retained cheaply for later ad-hoc analysis. What is the most cost-effective way to meet these requirements?

A payment service running on GKE writes structured JSON logs to Cloud Logging. During a partial outage, the on-call engineer needs to quickly find all ERROR-severity entries that contain a specific transaction ID field (jsonPayload.txnId) across the last 30 minutes, then aggregate how many distinct pods emitted them. The engineer wants the fastest interactive way to both filter and run aggregation without exporting data elsewhere. Which approach should they use?

A microservice on GKE writes plain-text log lines to stdout like 'ERROR user 4521 checkout failed timeout=3000ms'. The reliability team wants to filter these entries by severity in Logs Explorer and build a log-based metric that counts only checkout failures per user tier, but currently all entries appear with severity 'Default' and the message body is treated as a single unstructured string. What is the most effective change to enable reliable severity filtering and field-based extraction?

Your team runs a fleet of Compute Engine VMs hosting a legacy Java application. In Cloud Monitoring dashboards you can see CPU utilization and network traffic for each VM, but memory utilization and disk usage metrics are missing, and you cannot build a memory-pressure alert. The VMs use a standard public Debian image. What is the most likely cause and the correct fix?

A payments microservice runs on GKE and normally serves requests with p99 latency under 300ms. During brief, self-healing garbage-collection pauses, p99 latency spikes above 500ms for 20-40 seconds several times per hour. You created a Cloud Monitoring alerting policy on p99 latency with a threshold of 500ms, but the on-call team is being paged constantly for these transient spikes that recover on their own. You must reduce noisy pages without missing sustained latency degradation that actually harms users. What is the best change to the alerting policy?

Your team runs a fleet of 40 Compute Engine instances behind a managed instance group serving a REST API. You create a Cloud Monitoring alerting policy on the metric compute.googleapis.com/instance/cpu/utilization to page on-call when CPU is sustained above 85%. During a real incident where several individual instances spiked to 95% while the group average stayed near 60%, the alert never fired. You inspected the policy and found it aligns the series and then applies a cross-series reducer of 'mean' before evaluating the threshold. What change will make the alert correctly detect per-instance saturation?

A batch data-ingestion service writes a custom metric 'pipeline/records_processed' every 5 minutes when a cron-triggered job runs successfully. Recently the cron scheduler silently stopped firing, so the job never ran and no data points were written at all. Your existing threshold-based alert (fires when records_processed < 100) never triggered because the metric simply stopped reporting. You need an alerting policy that notifies the on-call engineer when the job stops producing the metric entirely. What should you configure?

A payments service on GKE occasionally leaks file descriptors, and the on-call team currently receives a Cloud Monitoring alert and manually runs a script to restart the affected Deployment. Your team wants to automatically trigger that remediation script (running as a Cloud Function) the moment the alert condition fires, while still keeping a human notified. The alerting policy already has an appropriate condition. What is the most appropriate way to wire the alert to the automated remediation?

A payments API team has an alerting policy in Cloud Monitoring that fires when the 5-minute p95 latency exceeds 800 ms. During a period of intermittent load, the underlying condition crosses the threshold repeatedly for short spikes, generating a storm of open-and-close incident notifications that overwhelm the on-call engineer. The team wants to keep the latency threshold but suppress noise from brief, transient spikes without missing a genuinely sustained latency regression. What should they configure?

Your team runs a batch data-processing job on Compute Engine that periodically computes the number of records that failed validation in each run. Operations wants this value visualized on a Cloud Monitoring dashboard and available for alerting, but the value is produced only by your application logic and is not exposed by any existing GCP metric or log entry. What is the most appropriate way to get this value into Cloud Monitoring?

Your team runs a high-traffic API on GKE. During an incident review, engineers complained that the existing latency dashboard—which plots the p50, p95, and p99 line charts from a distribution metric—hid a bimodal latency pattern where a subset of requests were consistently slow while most were fast. Leadership asks you to build a single dashboard widget that best reveals the full shape of the latency distribution over time so this pattern is immediately visible. Which Cloud Monitoring visualization should you configure?

Your team exports request latency from a Cloud Run service as a distribution (histogram) metric to Cloud Monitoring. The SLO states that 99% of requests must complete in under 400ms over a rolling 28-day window. During a review, a stakeholder asks how you can build a chart that shows the actual 99th-percentile latency trend so you can compare it against the 400ms threshold. Which approach correctly produces this on a Cloud Monitoring dashboard?

A payments microservice running on GKE writes structured JSON logs to Cloud Logging. The on-call team wants to be paged only when a specific critical error (jsonPayload.error_code = "PAYMENT_GATEWAY_TIMEOUT") occurs, and they want the notification delivered to their PagerDuty rotation within seconds of the log entry being written. They do not want to maintain a Prometheus scraper or write custom code. What is the most appropriate and efficient way to configure this?

Your team runs a payment API on GKE. The application already emits structured JSON logs to Cloud Logging that include a 'transaction_status' field, but the code was not instrumented with any custom metrics. Leadership wants a Cloud Monitoring dashboard tile showing the rate of failed transactions per minute, and they want it available as quickly as possible without a new application deployment. What is the most appropriate approach?

Your team runs a set of microservices on GKE that already expose application metrics in Prometheus exposition format on a /metrics endpoint. You need these custom metrics ingested into Cloud Monitoring so you can query them alongside GKE system metrics and build dashboards, without deploying and maintaining your own Prometheus server or storage. What should you do?

You run a fleet of 40 Compute Engine VMs behind a managed instance group serving a stateless API. You configure a Cloud Monitoring alerting policy on the CPU utilization metric with a threshold of 85% sustained for 5 minutes. During a load test, three individual VMs briefly exceeded 85%, but the aggregated policy never fired even though your team wanted to be notified about any single overloaded instance. Investigating the condition, you find the metric is being combined across all instances. Which change to the alerting condition will ensure you are alerted when any individual VM crosses the threshold?

Your organization runs a customer-facing application split across three separate GCP projects: 'frontend-prod', 'backend-prod', and 'data-prod'. The SRE team wants a single Cloud Monitoring dashboard that displays latency and error-rate charts combining metrics from all three projects, without duplicating dashboards or copying metric data. What is the recommended approach?