Question

A payments service on GKE occasionally leaks file descriptors, and the on-call team currently receives a Cloud Monitoring alert and manually runs a script to restart the affected Deployment. Your team wants to automatically trigger that remediation script (running as a Cloud Function) the moment the alert condition fires, while still keeping a human notified. The alerting policy already has an appropriate condition. What is the most appropriate way to wire the alert to the automated remediation?

Accepted Answer

Add a Pub/Sub notification channel to the alerting policy and configure the remediation Cloud Function with a Pub/Sub trigger on that topic, plus an email channel for the on-call team.

Answer

Configure the alerting policy to use only a Slack notification channel and have an operator manually invoke the Cloud Function from the Slack message.

Answer

Create a log-based metric on the alert and schedule the Cloud Function with Cloud Scheduler to poll that metric every minute.

Answer

Point the alerting policy's email notification channel at the Cloud Function's HTTP trigger URL so the email invokes the function.