Question

A payments microservice running on GKE writes structured JSON logs to Cloud Logging. The on-call team wants to be paged only when a specific critical error (jsonPayload.error_code = "PAYMENT_GATEWAY_TIMEOUT") occurs, and they want the notification delivered to their PagerDuty rotation within seconds of the log entry being written. They do not want to maintain a Prometheus scraper or write custom code. What is the most appropriate and efficient way to configure this?

Accepted Answer

Create a log-based alerting policy in Cloud Monitoring with a query matching jsonPayload.error_code="PAYMENT_GATEWAY_TIMEOUT", and attach a PagerDuty notification channel.

Answer

Create a counter log-based metric for the error, then build a metric-threshold alerting policy with a rolling 5-minute alignment window and a PagerDuty channel.

Answer

Export the logs to BigQuery with a log sink and schedule a query that runs every minute to detect the error and trigger a Cloud Function that calls the PagerDuty API.

Answer

Deploy a sidecar container that tails the pod logs and posts directly to PagerDuty whenever the error string appears.