What is the difference between HPA and VPA?

HPA (Horizontal Pod Autoscaler) adds and removes pod replicas based on observed metrics: typically CPU, memory, or a custom metric like requests per second. VPA (Vertical Pod Autoscaler) adjusts the CPU and memory requests of existing pods. HPA is the default choice; VPA is mostly used for stateful workloads where horizontal scaling isn't possible.

Why is CPU a bad autoscaling signal?

CPU usage lags real load: it spikes after queues have already filled up, which means the autoscaler reacts after users are already feeling the slowdown. Better signals include requests per second, queue depth, p99 latency and downstream saturation. KEDA was built specifically to autoscale on event-driven metrics like Kafka lag or AWS SQS queue length.

10 min read · Guide · Cloud · Operations

How it works · Operations

How autoscaling works across pods, nodes, and clusters.

Pods, nodes and clusters each scale on their own loop, against their own signal, with their own lag. Treating them as a single setting leads to over-provisioning or slow reactions.

Parts01–10 InteractiveMode picker PrereqK8s · metrics

What is autoscaling?

Autoscaling is the practice of adjusting capacity to match demand without manual intervention. In Kubernetes there are three layers: HPA (Horizontal Pod Autoscaler) adds and removes pods, VPA (Vertical Pod Autoscaler) resizes pod resources, and the Cluster Autoscaler adds and removes nodes. Each operates on different signals and at a different speed.

If you provision for peak you pay for idle capacity most of the day, and if you provision for average you fail on the spike. Autoscaling is a control loop that watches a signal, compares it to a target and adjusts capacity. The cost is operational complexity, since the loop has to be tuned, observed and trusted.

There are three loops. The pod-level loop (HPA) adds replicas of a Kubernetes pod. The workload-level loop (VPA) resizes CPU and memory requests. The cluster-level loop provisions new nodes when pods cannot schedule. Each runs independently and has its own lag.

The three layers: pod, node, and cluster

The three loops solve different problems, and many production systems run all three.

Horizontal scaling adds replicas of the same thing.

When CPU rises, add another pod or VM running the same image. The replicas are identical and a load balancer fans traffic out across them, so stateless workloads scale close to linearly. Startup is slow when the image has to be pulled and the app warmed up; pre-baked images and lazy init shorten it.

Why CPU is rarely the right autoscaling signal

HPA defaults to CPU because it is universally available and cheap to compute. But CPU is a poor proxy for whether the service is overloaded. A web server may pin CPU and still hold tail latency steady, while a worker may sit at 30% CPU and have minutes of work queued.

The best metric is the one that tracks degradation: queue depth for workers, p99 latency for user-facing services, and active connections for chat and streaming. Use a custom metrics adapter (Prometheus + KEDA, Datadog, etc.) to expose them to HPA.

# HPA targeting custom queue-depth metric
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
 minReplicas: 2
 maxReplicas: 50
 metrics:
 - type: External
 external:
 metric:
 name: rabbitmq_queue_depth
 selector: { matchLabels: { queue: orders } }
 target: { type: AverageValue, averageValue: "30" }
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 300 # avoid yo-yo
 policies: [{ type: Percent, value: 25, periodSeconds: 60 }]

Why scaling up takes more than a minute

HPA reads metrics every 15 seconds and acts on a window. New pods take 10–60 seconds to be ready (container image pull, app warm-up, JIT, connection-pool init). Cluster autoscaler is much slower: 60–120 seconds to bring a new node online. End to end, a sudden 4× traffic spike can hurt for 90+ seconds before steady state returns. A load balancer in front of the pods determines how that ramp distributes.

Three levers reduce the delay: headroom (run more replicas than peak demand at all times), predictive scaling (schedule changes ahead of known patterns), and burstable capacity (Karpenter, AWS Fargate Spot). Autoscaling alone will not absorb an instant 5× spike, so pair it with pre-warmed capacity.

A lunchtime spike, minute by minute

Consider a lunchtime spike. At 11:58 office workers open the app and traffic starts climbing; by 12:04 it has doubled. The pods feel it right away, with CPU rising and request queues building, but the autoscaler doesn't. Metrics-server scrapes kubelets on its own interval, and HPA reads the aggregated result on a 15-second sync loop. The number HPA acts on at 12:04 describes the cluster as it was up to half a minute earlier.

At 12:05 the arithmetic finally crosses the threshold and HPA asks for six more replicas. Scale-up has no stabilization window by default, so the request goes out at once, but the scheduler can only place two of the six. The other four sit Pending: every node is full. HPA adds pods, not machines, so it can only wait for new nodes.

The cluster autoscaler notices the Pending pods on its next scan (every ten seconds), picks a node group, and asks the cloud for a VM. The instance boots, joins the cluster and pulls images, which takes 60 to 120 seconds or more if the image is heavy. The four pods schedule at 12:07, warm up, and pass readiness around 12:08. Lunch traffic peaked at 12:06, so the new capacity arrives in time to serve the decline.

Each of these settings changes a different part of that timeline, and none of them makes capacity arrive earlier:

Stabilization window: HPA's stabilizationWindowSeconds. On scale-up it defaults to zero: raise it and the 12:05 reaction waits out the window first, trading speed for smoothness. On scale-down it defaults to 300 seconds, which is why the extra replicas hold through the noisy 12:30 lull instead of collapsing on the first quiet scrape.
Cooldown: The pause after one scaling action before the next (ASG cooldowns; HPA's periodSeconds policies). At 12:06, when six replicas turn out not to be enough, cooldown decides how long the second correction has to wait behind the first.
Scale-down delay: Cluster autoscaler's scale-down-delay-after-add and scale-down-unneeded-time (about ten minutes by default). They decide how long the new node survives after traffic falls. If it is too short the node is gone before the 13:00 stragglers, and if it is generous the node is still there for them at the cost of idle capacity in between.

The lag itself (scrape interval plus sync loop plus boot time) is fixed, and these settings only shape behaviour around it. Since lunch happens at the same hour every day, a scheduled scale-up at 11:45 beats any reactive tuning above, which is the case for predictive scaling on spikes you can see coming.

Horizontal pod autoscaling (HPA) and flapping

A classic autoscaling problem: scale up because CPU is high, the new pod absorbs load, CPU drops below the threshold, scale down, and repeat. Pods churn, cold starts hurt, and the replica count never settles. A warm cache on the new pod is what closes the gap.

Fixes include a wider band between scale-up and scale-down thresholds, a stabilization window (HPA stabilizationWindowSeconds), or a different metric that doesn't dip immediately on adding capacity (queue depth holds for a beat; CPU drops instantly).

Vertical pod autoscaling (VPA) and the restart it costs

VPA recommends new CPU and memory requests for your pods based on actual usage. Recommendation mode is safe; auto mode rolls pods to apply the change. That means restarts, and VPA in auto mode on a stateful workload can restart pods at peak load.

Most teams run VPA in recommendation mode and apply the suggestions in the next deploy. Right-sizing avoids both OOM kills and waste, but the unplanned restarts of auto mode often outweigh that.

Cluster autoscaling and Karpenter

Cluster Autoscaler scales pre-defined node groups: fixed instance type, fixed shape. Karpenter (AWS, then ported elsewhere) takes a different approach: it watches pending pods, picks the best-fit instance type from a wide catalog, and provisions it directly. No node groups.

It is faster because it skips the ASG step, and it fits better because it picks the right shape per workload. Many AWS Kubernetes teams have moved to Karpenter for this reason, since the right-sizing savings can pay for the migration.

Better signals: RPS, queue depth, and p99 latency

CPU lags real load. Better signals, with the systems that expose them:

Requests per second (RPS): A good signal for stateless web services. Scale when RPS-per-pod exceeds the capacity you measured under load testing. Exposed by ingress controllers (NGINX, Envoy) and by the Kubernetes external metrics API. KEDA's prometheus scaler is the easiest way to wire this up.
Queue depth: For workers consuming a queue (Kafka, SQS, RabbitMQ): the signal is consumer lag (Kafka), the visible message count (SQS) or the queue depth (RabbitMQ). KEDA ships scalers for all three, and scaling on queue depth tracks the producer rate closely.
p99 latency: When the application's perceived performance matters more than its CPU usage. Scale when p99 > target_latency. This is trickier than RPS because latency only means something at a steady request rate, so it is usually a secondary signal alongside RPS.
Concurrent connections: For long-lived WebSocket / SSE servers, since RPS is meaningless. Memory per connection × headroom = concurrent connection target. Scale at 70% of saturation to leave room for new arrivals.

KEDA. KEDA (Kubernetes Event-Driven Autoscaler) is the most widely used option for non-CPU autoscaling on Kubernetes. It runs as a separate operator that watches an external metric (Kafka lag, SQS depth, Prometheus query, AWS CloudWatch alarm) and feeds it to the standard Kubernetes HPA via the external metrics API. It ships with more than sixty prebuilt scalers, and the scalers catalog has the full list.

How big sites autoscale in production

Pinterest: KEDA + Karpenter. Pinterest scales ~5,000 microservices, mostly via KEDA on RPS metrics from Envoy, with Karpenter (rather than the cluster autoscaler) provisioning nodes. Their public engineering writeups report ~30% cost savings from switching to Karpenter for spiky workloads, and ~40% faster scale-up than the legacy cluster autoscaler.

Stripe: predictive scaling. Stripe's API tier autoscales on a combination of inbound RPS and a 7-day rolling forecast. The forecast component handles predictable spikes (Black Friday, payday Mondays in major timezones) by pre-warming capacity. Stripe described the architecture in its engineering writeups on API traffic forecasting; the predictor uses gradient-boosted decision trees on hour-of-week and recent-trend features.

AWS Lambda: invocation-driven, with no separate autoscaler. Lambda doesn't use HPA-style autoscaling. Every concurrent invocation gets its own execution environment (a Firecracker microVM), so 1,000 simultaneous requests can mean 1,000 cold starts, which provisioned concurrency mitigates for latency-sensitive paths. The model differs from HPA but addresses the same problem.

A closing note

Autoscaling is a control system, and control systems can oscillate. Pick a metric that tracks what hurts users, set wide bands, and reserve headroom for spikes.