How a container is just a process with its own view of the system.

A container is a process running on a host kernel, isolated from other processes by Linux namespaces (PID, network, mount, UTS, IPC, user, cgroup, time) and resource-bounded by cgroups. Docker (2013), then the OCI standard (2015), made containers ubiquitous; Kubernetes made them schedulable. Containers are not VMs: they share the host kernel.

A container is a regular Linux process. It runs on the same kernel as everything else on the host. What makes it look isolated is two kernel features applied to that process at startup: namespaces (which kernel state the process can see) and cgroups (which resources it may consume).

Docker, containerd and OCI are conventions about how to set up the namespaces, package the filesystem and ship the image. Every "container runtime" calls the same handful of clone(), unshare(), and setns() syscalls.

Each namespace isolates one kernel resource. Pick a namespace below to see what it changes inside the container.

A namespace isolates the view, and a cgroup bounds the resource consumption (CPU shares, memory ceilings, block I/O). Together they make a container's behaviour on the host predictable.

cgroup v2 vs v1. The original cgroup v1 (kernel 2.6.24, 2008) used a separate hierarchy per controller, so a process could sit in different cgroups for CPU and memory. That was flexible but hard to operate. cgroup v2 (kernel 4.5, 2016, default in modern distros since 2021) unifies them: one tree, one cgroup per process across all controllers. cgroup v2 support went GA in Kubernetes 1.25, and Docker 20.10+ supports both versions. The memory.high attribute in v2 (a soft limit that throttles instead of killing) is a big improvement over v1's hard-only limits.

Real numbers. A typical Kubernetes pod spec sets resources.limits.cpu: 1000m (one full core) and resources.limits.memory: 2Gi. The kubelet translates this to cpu.max=100000 100000 (CFS quota: 100ms of CPU per 100ms period) and memory.max=2147483648 in the cgroup. When the container exceeds memory.max it gets OOM-killed by the kernel; when it exceeds CPU it gets throttled (run at slower wall-clock pace, no kill). That difference is why memory limits hurt more than CPU limits.

The CFS quota throttling problem. A process that bursts to 100% CPU for 50ms, then idles 50ms, runs fine on a bare host but gets throttled inside a 1-CPU cgroup because CFS counts wall-clock time across all threads. A burst of 8 threads at 100% for 12.5ms uses up the 100ms quota in one tick. Production fix: either set --cpu-period=10ms (smaller quota window) or grant more CPU than the average load suggests. Many teams have a rule of thumb that "CPU limits are evil" because of this.

A container image is a stack of read-only filesystem layers plus a writable top layer: cached by hash on every host that pulled them once. Each Dockerfile instruction (RUN apt install, COPY src) creates a new layer; the layers below are shared with every other image that uses them. This is why images built FROM ubuntu share most of their disk space.

At runtime, an overlay filesystem stacks the layers; writes go to the top (writable) layer using copy-on-write. Two containers from the same image share their entire read-only stack until one writes, at which point the changed file is copied into that container's top layer.

The Open Container Initiative defines three specs that decouple "what runs the container" from "what builds the image": the image format, the runtime spec (what bundle a runtime accepts and how it should configure namespaces / cgroups / mounts), and a distribution API (how registries push and pull). Docker's image format is OCI's. Podman, containerd, cri-o all use OCI runtimes (typically runc).

This is what made Kubernetes possible: pluggable container runtimes through a single interface. Runc can be swapped for Kata (VM-isolated), gVisor (userspace kernel) or youki (a Rust port) with the same image and config but different sandbox properties.

A virtual machine boots a full operating system on virtual hardware: its own kernel, drivers, init system. It is heavy, but the security boundary is the hypervisor, which is small and well audited.

A container shares the host kernel. It is light (starts in milliseconds, megabytes of overhead instead of gigabytes), but the security boundary is the kernel itself, a much larger surface. Container escapes have happened (the runc /proc/self/exe overwrite, CVE-2019-5736; the runc leaky-fd escape, CVE-2024-21626). For untrusted code, VM-level isolation (Kata, Firecracker, gVisor) can sit below the container API, giving container-style startup with something closer to VM isolation.

A container is a process. It still suffers from noisy-neighbour effects on the host (CPU steals, I/O contention) unless cgroups are tuned aggressively: autoscaling hides the symptoms without fixing the cause. Persistent state needs explicit volume mounts; the writable layer is gone when the container exits. Logging and metrics need pipes to host-side collectors. Secrets belong outside the image, not baked in.

When something works in a container locally but fails in production, the cause is usually one of the four above.

The container security boundary is the Linux kernel: every container shares it. A VM's boundary is the hypervisor (~50,000 lines of code in Firecracker's case), while the Linux kernel is ~30 million lines, so the attack surface is far larger. Notable public container-escape CVEs:

CVE-2019-5736 · runc

An attacker with write access to the host's runc binary could overwrite it via /proc/self/exe. Patched in days; affected almost every container runtime.

CVE-2022-0185 · Linux kernel

Heap overflow in fs/fs_context.c. A privileged container could trigger it to escape to the host. Fixed in 5.16.2.

CVE-2024-21626 · runc "Leaky Vessels"

File descriptor leak let a container's WORKDIR reference escape into the host filesystem. Affected Docker, containerd, Podman.

CVE-2022-23648 · containerd

Symlink-following bug let a container read arbitrary files from the host via volume mounts.

Defense in depth. The kernel boundary should not be the only control. Typical layers are seccomp profiles (whitelist syscalls, Docker's default blocks ~50 of them); AppArmor or SELinux (mandatory-access-control); read-only root filesystem (--read-only); drop capabilities (--cap-drop=ALL); non-root user in the image (USER 1000); and no privileged mode. CIS Docker Benchmark and Kubernetes Pod Security Standards encode these.

For untrusted code, use VM-isolated runtimes. Kata Containers wraps each container in a lightweight VM (~10MB overhead, ~200ms boot). gVisor intercepts syscalls in user space (the "Sentry"), trading some performance for kernel-attack-surface reduction. Firecracker is what AWS Lambda and Fargate run on: micro-VMs in ~125ms with ~5MB memory footprint. All three plug into the OCI runtime interface, so Kubernetes can run them via runtimeClassName.

The runtime stack has two layers. The high-level runtime (also called the "container engine") manages images, networking, volumes: Docker, containerd, CRI-O, Podman live here. The low-level runtime creates the namespaces and cgroups: runc is the dominant one; crun and youki are alternatives.

Docker Engine

The original. Includes a CLI, a daemon, image build, BuildKit. Dropped from Kubernetes 1.24 (May 2022) when the dockershim adapter was removed; Docker Desktop and standalone Docker are still widely used for local development.

containerd

Born from Docker, donated to CNCF in 2017, graduated 2019. The default runtime in EKS, GKE, AKS, kind, and most managed Kubernetes today. Lighter than Docker (no image build, no built-in networking), exposes the CRI directly to kubelet.

CRI-O

Red Hat's CRI-only runtime: deliberately the narrowest scope of the four. Default in OpenShift. Tighter security defaults than containerd; smaller surface.

Podman

Daemonless (no background process), drop-in CLI compatibility with Docker. Strong on rootless containers (a regular user can run Podman without sudo). Default on RHEL 8+ and Fedora.

Real-world picks. Build images on a developer laptop: Docker Desktop or Podman. Run containers in a Kubernetes cluster: containerd or CRI-O via the kubelet. Run untrusted user code (CI runners, sandboxed function execution, multi-tenant SaaS): Kata or gVisor under containerd, or Firecracker if you can use AWS infrastructure. Because of the OCI standard, the choice is mostly operational and rarely affects the architecture. For the cluster-side view, where the kubelet drives the CRI from pod spec to running container, see Kubernetes internals.

You type docker run nginx. The CLI doesn't run anything: it POSTs to the Docker daemon over a Unix socket, and the daemon delegates to containerd. containerd checks its local content store for the nginx image. Not there, so it talks to the registry: fetch the manifest for the latest tag, read the list of layer digests, then download each layer blob it doesn't already have, verifying every one against its sha256. Each layer unpacks into its own snapshot directory under /var/lib/containerd.

Next, the rootfs. containerd mounts an overlay filesystem: the read-only layers become lowerdir entries, a fresh empty directory becomes upperdir, and the merged view appears at a single mount point. That merged directory is the container's root filesystem. Nothing was copied: a second nginx container would reuse every lower layer and get only its own empty upperdir.

containerd writes an OCI bundle, a config.json next to that rootfs, and hands it to a shim process, which invokes runc. runc calls clone() with CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWUTS | CLONE_NEWIPC. The child is born into fresh namespaces: it sees an empty process table and a network stack with nothing but loopback. The runtime writes the child's PID into a new cgroup directory and sets memory.max and cpu.max from whatever --memory and --cpus flags you passed. From that write onward the kernel enforces the limits; the runtime's job there is done.

Still inside the child, before anything from the image runs: mount /proc and /dev, then pivot_root onto the overlay's merged directory. pivot_root swaps the root mount and unmounts the old one, so the host's filesystem isn't hidden, it's unreachable. Then the runtime drops capabilities, applies the seccomp profile, and calls execve() on the image's entrypoint: nginx -g 'daemon off;'. The exec replaces the setup code in place, so nginx inherits the namespaces, the cgroup membership, and a root it can't see out of. Inside its PID namespace it is PID 1, with PID 1's duties: reap zombies, handle signals. On the host it's just process 48-thousand-something.

Meanwhile the daemon wired up networking: a veth pair with one end inside the container's net namespace as eth0 and the other plugged into the docker0 bridge, plus a NAT rule if you passed -p. Total elapsed: maybe 300 ms warm, most of it image pull when cold. No kernel was booted. One process was created with unusual flags, confined, and replaced itself with nginx.

Netflix (Titus, ~3 million container starts per day). Built their own scheduler on top of Mesos + their own container runtime, then migrated to Kubernetes-on-Titus in 2024. Run a mix of containerd (most workloads) and Firecracker (for untrusted creator-uploaded code in their VFX pipeline). Memory limits are mandatory, and an OOM kill pages someone.

AWS Lambda (trillions of invocations a month). Each function invocation runs in a Firecracker microVM: ~125 ms cold start, ~5 MB memory footprint per VM, ~thousands of VMs per host. The multi-tenant boundary has to survive arbitrary kernel exploits, and Firecracker's small attack surface (~50,000 LOC) meets that bar where a Linux container alone would not.

GitHub Actions runners. Public-cloud GitHub-hosted runners spin up a fresh VM per job (Azure VM, ~20-second boot). Self-hosted runners run inside containers (containerd, often with Kubernetes' Actions Runner Controller). The trade-off is the security boundary: untrusted PR code on a hosted runner gets a VM, while trusted internal code can run in a container.