Question

Your team runs a payment API on GKE. The application already emits structured JSON logs to Cloud Logging that include a 'transaction_status' field, but the code was not instrumented with any custom metrics. Leadership wants a Cloud Monitoring dashboard tile showing the rate of failed transactions per minute, and they want it available as quickly as possible without a new application deployment. What is the most appropriate approach?

Accepted Answer

Create a counter log-based metric that filters log entries where transaction_status="failed", then add a chart of that metric's rate to the dashboard.

Answer

Add OpenTelemetry custom metric instrumentation to the application code and redeploy so the failure counter is exported directly to Cloud Monitoring.

Answer

Configure Managed Service for Prometheus to scrape a /metrics endpoint on the pods and build the dashboard from the scraped counter.

Answer

Export the logs to BigQuery via a log sink and schedule a query that counts failures, then visualize the result in Looker Studio.