Kubernetes pod eviction simulator

List the pods on one node with their priority and memory request, limit and current usage, and set the node's memory, what the system and kubelet use, and the hard eviction threshold. The simulator works out each pod's QoS class and oom_score_adj, applies container limits, then the kubelet's eviction if memory.available is below the threshold, and the kernel's OOM killer if memory runs out entirely. Add pdb lines to see what kubectl drain would do.

Kubernetes puts every pod into a quality of service class from its containers' resource requests and limits. A pod is Guaranteed if every container has CPU and memory limits equal to its requests, BestEffort if no container has any request or limit, and Burstable otherwise. If a container sets a limit but no request, the request defaults to the limit. The class is not a priority setting: it describes how much the pod has promised about its use, and Kubernetes uses it to decide whose processes the kernel should kill first.

The kubelet watches eviction signals such as memory.available. On Linux its default hard thresholds are memory.available<100Mi, nodefs.available<10%, imagefs.available<15% and nodefs.inodesFree<5%. When a signal crosses its threshold, the kubelet ranks the pods and evicts them, re-measuring after each, until the signal recovers. For memory the ranking is:

1. pods whose usage exceeds their request come first (BestEffort pods always do);
2. then lower priority before higher;
3. then the pod furthest above its request.

In the Memory pressure example, with a 500Mi threshold, memory.available is 44Mi. The ranking is batch-1, reports-1, web-1, cache-1, web-2, payments-1; the kubelet evicts batch-1 and then reports-1, after which memory.available is 1.02Gi. The payments pod, which stays within its request, would be evicted last; Guaranteed pods are only evicted when system processes, not other pods, eat into the node's memory. Hard-threshold evictions ignore the pod's grace period and any PodDisruptionBudget.

The kubelet checks memory periodically, so a sudden spike can exhaust the node first. Then the Linux kernel's OOM killer picks a process to kill by its badness, roughly its share of memory in thousandths plus its oom_score_adj, the number shown in /proc/<pid>/oom_score. Kubernetes sets oom_score_adj from the QoS class: −997 for Guaranteed, so they are killed almost never; 1000 for BestEffort; and for Burstable, 1000 − 1000 × request / node memory, kept between 2 and 999, so a larger request means better protection.

The kernel and the kubelet do not agree. In the memory pressure example the highest oom_score belongs to cache-1 (1112), because it uses the most memory, while the kubelet evicts batch-1 first, because it is the lowest-priority pod over its request. In the Out of memory example the pods want more than the node has (memory.available −240Mi), and the kernel kills etl-1.

A container that goes over its own memory limit is killed by the kernel inside its cgroup, whatever the rest of the node looks like. The pod shows OOMKilled with exit code 137 (128 plus signal 9), and the kubelet restarts the container in place, with back-off if it keeps happening. In the Over its own limit example, api-1 is OOMKilled while the node has plenty of memory free.

An Evicted pod was removed whole by the kubelet because the node was short of a resource; its status says "The node was low on resource: memory." The pod does not restart on that node; its controller creates a new one, which the scheduler may place elsewhere. Frequent evictions usually mean requests are set below real use, so the scheduler packs more onto nodes than they can hold.

Evictions for maintenance are voluntary disruptions. kubectl drain cordons the node, then asks the Eviction API to evict each pod. The API checks the pod's PodDisruptionBudget: if evicting it would leave fewer healthy pods than minAvailable (or more unavailable than maxUnavailable), it refuses with HTTP 429, and drain retries. In the Drain with PDBs example, web-1 and batch-1 are evicted and web-2 and zk-1 are blocked until replacements are healthy elsewhere. A budget that can never be met, such as minAvailable equal to the replica count, blocks drains forever.

Each pod is treated as one container with one memory figure; CPU is assumed to follow the same request and limit pattern when the QoS class is worked out. A container killed for exceeding its limit is counted as using nothing while it restarts. Memory pressure is evaluated once, with the usage you give, rather than over time; soft thresholds, eviction-minimum-reclaim (0 by default) and node-level reclaim before evicting pods, such as freeing unused images for disk pressure, are left out.