What happens between kubectl apply and a running container

Kubernetes pod creation is an eleven-step dance between the API server, etcd, the scheduler, the kubelet, and the container runtime. From kubectl apply to a running container takes typically 5-30 seconds. Each step is a separate watch-driven event; understanding the sequence is the foundation of debugging Kubernetes.

Kubernetes' design is deceptively flat. There are only four kinds of components you need to know to understand pod creation: the API server, etcd, a fleet of controllers (the scheduler being one), and per-node kubelets. None of them call each other directly. Every interaction goes through the API server.

Every controller in Kubernetes opens a streaming HTTP connection to the API server with ?watch=1 and stays connected forever. Each event — ADDED, MODIFIED, DELETED — streams down as it happens. Restart a controller? It does a full LIST first to bootstrap, then resumes the watch from the last resourceVersion it saw.

This is what makes Kubernetes feel reactive instead of polled. When the scheduler assigns a pod, the kubelet on that node knows within milliseconds — not because anyone told it, but because it was already watching, and the API server pushed the change down its open connection.

The simulator below plays a single kubectl apply -f pod.yaml from the command landing on the API server through the container running on node-3. Five lanes, eleven hops. Use + Show wire bytes to see the exact payload at each step.

The default scheduler is a small, well-defined program. Every unscheduled pod goes through two passes: filter (predicates that say no), then score (priorities that say better-or-worse). The highest-scoring node wins.

This shape — filter then score — is what makes the scheduler pluggable. You can write a custom scheduler plugin and add it to either pass. The scheduling framework calls every plugin in order; one plugin's no halts filtering, but every plugin contributes to scoring.

Kubelet is just another controller — it watches pods, reconciles to the desired state. What's special is what reconcile means here: it has to make Linux processes happen. The kubelet itself never starts a container; it tells the container runtime to.

The runtime — containerd, cri-o, historically Docker — speaks a gRPC interface called the Container Runtime Interface. Behind that interface, runc (or kata, gvisor, youki) actually clone()s the process into a fresh set of Linux namespaces, applies cgroups, mounts the rootfs, and execs the container command.

The eleven steps you just walked through are not a procedure. None of them is "called" by anything. Each component watches some piece of state and does its small job whenever the watched state changes. Pod creation is just one well-defined trajectory through that watch graph.

When you delete a pod: kubelet sees the deletion event, gracefully terminates the container, removes the network attachment, deletes the sandbox. When a node dies: the node-lifecycle controller sees the heartbeat stop, marks the node NotReady, and eventually evicts the pods, which the deployment controller notices and recreates, which the scheduler picks up — same loop, different starting state.

The eleven-hop trace is also a debugging map. When a pod is "not running," it has stalled at one of those steps — and each has a recognizable signature.

The control plane is not magic. It is a watch loop, a scheduler, and a per-node reconciler — and the watch loop is open to you. Define a Custom Resource Definition, write a controller that watches it, and reconciles toward the spec, and you have just extended Kubernetes.

This is the operator pattern. cert-manager watches Certificate CRs and provisions TLS via ACME. prometheus-operator watches ServiceMonitor CRs and updates Prometheus configs. The Postgres operator watches Postgres CRs and runs a HA cluster. Every one of them is the same shape as the pod flow you just walked through.

If you want to deepen this, see the K8s networking guide for what happens after the pod is running.

From kubectl apply to the container's first byte served, the typical pod takes 5-30 seconds. Where the time goes:

API server admission + etcd write

~50-200ms. Validates the spec, runs admission webhooks, writes to etcd, broadcasts the watch event.

Scheduler decision

~100-500ms. Filter and score nodes. Typically the fastest hop; bottlenecks here usually mean an expensive admission webhook or scheduler plugin.

Image pull

~1-30s. Dominant for cold image cache. Can be cut with image-streaming (containerd's lazy image-pull, AWS Soci) or by warming the local cache via DaemonSet.

Container runtime sandbox creation

~100-500ms. The pause container, the network namespace, the cgroup tree, the storage mount. Fast unless you have many init containers.

CNI plugin

~50-300ms. Allocate the pod IP, plumb the routes. AWS VPC CNI was historically slow (~1-2s) on cold ENIs; prefix-delegation (2021) cut this to ~100ms.

Container start

~50ms-N seconds. Depends entirely on application startup. JVM cold start ~5-10s; Go binary ~50ms; Python with imports ~200-500ms.

Readiness probe success

~periodSeconds × failureThreshold. Default ~30s before kubelet considers the pod Ready, even if the container started in 50ms. The most common 'why is my deploy slow' answer.

Scale issues. At thousands-of-pods rolling deploys, etcd's update rate and the kubelet's image-pull throughput become bottlenecks. EKS/GKE clusters >5,000 nodes routinely measure pod-start times in minutes during big rollouts. Mitigations: pre-pull images via DaemonSet, use container-runtime image streaming, cap image size aggressively (Distroless and Alpine help).