What happens between kubectl apply and a running container

Kubernetes pod creation is a sequence of eleven events involving the API server, etcd, the scheduler, the kubelet, and the container runtime. From kubectl apply to a running container typically takes 5–30 seconds. Each step is a separate watch-driven event, so knowing the sequence helps a lot when debugging.

Kubernetes' design is fairly flat. Four kinds of components matter for pod creation: the API server, etcd, a fleet of controllers (the scheduler being one), and per-node kubelets. They do not call each other directly, and every interaction goes through the API server.

Every controller in Kubernetes opens a streaming HTTP connection to the API server with ?watch=1 and stays connected forever. Each event (ADDED, MODIFIED, DELETED) streams down as it happens. Restart a controller? It does a full LIST first to bootstrap, then resumes the watch from the last resourceVersion it saw.

Because of this, Kubernetes reacts to changes quickly without polling. When the scheduler assigns a pod, the kubelet on that node usually learns of it within milliseconds. Nobody tells it; it was already watching, and the API server pushed the change down its open connection.

The simulator below plays a single kubectl apply -f pod.yaml from the command landing on the API server through the container running on node-3. It has five lanes and eleven hops. Use + Show wire bytes to see the payload at each step.

Every unscheduled pod goes through two passes in the default scheduler: filter (predicates that rule nodes out), then score (priorities that rank the rest). The highest-scoring node wins.

Splitting the work into filter then score makes the scheduler pluggable. You can write a custom scheduler plugin and add it to either pass. The scheduling framework calls the plugins in order. A single plugin rejecting a node ends filtering for that node, while every plugin contributes to scoring.

The kubelet is another controller: it watches pods and reconciles toward the desired state. Here, reconciling means making Linux processes exist. The kubelet does not start a container itself; it tells the container runtime to.

The runtime (containerd, cri-o, historically Docker) speaks a gRPC interface called the Container Runtime Interface. Behind that interface, runc (or kata, gvisor, youki) clone()s the process into a fresh set of Linux namespaces, applies cgroups, mounts the rootfs, and execs the container command.

The eleven steps above are not a procedure, and nothing "calls" them in order. Each component watches some piece of state and does its small job whenever that state changes. Pod creation is one path through that watch graph.

When you delete a pod, the kubelet sees the deletion event, gracefully terminates the container, removes the network attachment and deletes the sandbox. When a node dies, the node-lifecycle controller sees the heartbeat stop, marks the node NotReady, and eventually evicts the pods. The deployment controller notices and recreates them, and the scheduler places them. It is the same loop with a different starting state.

The eleven-hop trace is also a debugging map. When a pod is "not running," it has stalled at one of those steps, and each stall has a recognizable signature.

The control plane is a watch loop, a scheduler and a per-node reconciler, and the watch mechanism is available to your own code. Define a Custom Resource Definition, write a controller that watches it and reconciles toward the spec, and you have extended Kubernetes.

This is the operator pattern. cert-manager watches Certificate CRs and provisions TLS via ACME. prometheus-operator watches ServiceMonitor CRs and updates Prometheus configs. A Postgres operator watches Postgres CRs and runs an HA cluster. Each follows the same pattern as the pod flow above.

For what happens after the pod is running, see the K8s networking guide.

From kubectl apply to the container's first byte served, the typical pod takes 5-30 seconds. Where the time goes:

API server admission + etcd write

~50-200ms. Validates the spec, runs admission webhooks, writes to etcd, broadcasts the watch event.

Scheduler decision

~100-500ms. Filter and score nodes. Typically the fastest hop; bottlenecks here usually mean an expensive admission webhook or scheduler plugin.

Image pull

~1-30s. Dominant for cold image cache. Can be cut with image-streaming (containerd's lazy image-pull, AWS Soci) or by warming the local cache via DaemonSet.

Container runtime sandbox creation

~100-500ms. The pause container, the network namespace, the cgroup tree, the storage mount. Usually fast, unless there are many init containers.

CNI plugin

~50-300ms. Allocate the pod IP, plumb the routes. AWS VPC CNI was historically slow (~1-2s) on cold ENIs; prefix-delegation (2021) cut this to ~100ms.

Container start

~50ms-N seconds. Depends entirely on application startup. JVM cold start ~5-10s; Go binary ~50ms; Python with imports ~200-500ms.

Readiness probe success

~initialDelaySeconds + periodSeconds. The pod becomes Ready after the first passing probe, so a large initialDelaySeconds or periodSeconds adds directly to rollout time even if the container started in 50ms. A common reason a deploy is slower than expected.

Scale issues. In rolling deploys of thousands of pods, etcd's update rate and the kubelet's image-pull throughput become bottlenecks, and pod-start times in clusters of more than 5,000 nodes can stretch to minutes. Mitigations: pre-pull images via DaemonSet, use container-runtime image streaming, cap image size aggressively (Distroless and Alpine help).