Question

A payments API running on GKE writes structured JSON logs to Cloud Logging. Each request log entry contains a field 'latencyMs'. The SRE team wants to alert when the 95th-percentile request latency exceeds 400 ms over a 5-minute window, but the application does not export any native Cloud Monitoring metrics. What is the most appropriate way to achieve this with minimal application changes?

Accepted Answer

Create a distribution-type log-based metric that extracts the 'latencyMs' field as the metric value, then build a Cloud Monitoring alerting policy using the 95th percentile aligner on that metric.

Answer

Create a counter-type log-based metric that counts log entries where 'latencyMs' > 400, then alert when the count is greater than zero.

Answer

Export all logs to BigQuery via a log sink and schedule a query that computes the 95th percentile, sending an email when the threshold is exceeded.

Answer

Instrument the application with the OpenTelemetry SDK to push a custom distribution metric and alert on the percentile in Cloud Monitoring.