Question

An SRE team spends roughly 12 hours per week manually restarting a stateful worker service whenever it exhausts memory and stops processing jobs. Each restart is identical: an on-call engineer receives a page, SSHes to the VM, runs a known command, and verifies queue drain. Leadership asks the team to reduce toil in line with SRE principles while preserving reliability. Which approach best addresses the toil according to Google SRE practice?

Accepted Answer

Write a script triggered by a memory-threshold alert that automatically performs the restart and verification, then invest the freed time in fixing the underlying memory leak.

Answer

Add more on-call engineers to the rotation so the 12 hours of restarts are distributed across more people and each person experiences less toil.

Answer

Increase the VM machine type to a larger memory size so restarts happen less frequently, and document the manual restart procedure in a runbook.

Answer

Lower the service SLO so the memory-related failures no longer count as reliability violations, reducing pressure to respond quickly.