Question

A payment-processing microservice runs on GKE and consumes jobs from a Pub/Sub subscription. During promotional events the subscription's unacknowledged message backlog grows to hundreds of thousands, causing job latency to spike, even though pod CPU utilization stays below 40%. The current Horizontal Pod Autoscaler scales only on CPU, so it rarely adds replicas during these events. You must ensure the service scales out to drain the backlog quickly while keeping steady-state cost low. What should you do?

Accepted Answer

Configure the HPA to scale on an external metric derived from the Pub/Sub subscription's num_undelivered_messages, using Custom Metrics Stackdriver Adapter, with a target value per pod.

Answer

Increase the HPA's minReplicas to a high fixed value so enough pods are always available to absorb promotional spikes.

Answer

Lower the HPA CPU utilization target to 20% so the autoscaler reacts sooner during promotional events.

Answer

Enable Vertical Pod Autoscaler in Auto mode to increase CPU and memory requests on the existing pods during backlog growth.