Question

Your team runs a payment API and has defined an availability SLO measured from server-side HTTP response codes at the load balancer. During a recent incident, users reported widespread payment failures for nearly 20 minutes, yet your SLI dashboard showed availability well within the SLO the entire time. Investigation revealed the load balancer returned HTTP 200 responses containing application-level error payloads generated when a downstream fraud-check dependency timed out. What is the most appropriate change to make your SLI reflect the actual user experience?

Accepted Answer

Redefine the 'good event' criteria for the SLI to count only responses that are both HTTP 200 and contain a successful application-level status, measured as close to the user as possible.

Answer

Lower the SLO target to a smaller number of nines so that the error budget is consumed faster and alerts fire sooner during similar incidents.

Answer

Add more load balancer instances and enable connection draining so that timeouts to the fraud-check dependency no longer occur.

Answer

Switch the SLI measurement source from the load balancer to raw CPU and memory utilization metrics of the payment API pods.