Thread pools, the boring workhorse.

A thread pool is a fixed set of worker threads that pull tasks off a shared queue and run them. The pattern dates to the 1990s; modern implementations include Java's ExecutorService, Go's goroutine scheduler, Rust's Tokio runtime, and the kernel's workqueue subsystem. Sizing the pool is the dominant lever: CPU-bound work sizes near NumCPU, I/O-bound work sizes much higher.

A thread pool decouples how work arrives from how work runs. Producers push tasks onto a queue; a fixed number of long-lived worker threads loop, dequeue, run, repeat. No thread is created or destroyed per task — that cost (typically ~100 μs to spawn, plus stack allocation) is paid once, at startup.

The trick is bounding. With unbounded workers your server keeps making more threads under load until it OOMs or thrashes. With unbounded queues you reply slowly and never push back. Both bounds are mandatory if you want a server that fails predictably — the same shape as a ring buffer with backpressure.

Submit jobs of varying size. The dispatcher hands them to free workers; busy workers count down. Notice the queue grows when arrivals exceed pool capacity — that is back-pressure forming, in real time. (See rate limiting for the upstream version.)

The size formula depends on what the workers do. For CPU-bound work (compression, parsing, math) you want N ≈ number of cores. More than that and threads time-share, adding context-switch overhead without finishing more work.

For I/O-bound work (database, HTTP, disk) the worker spends most of its time waiting. You want N to be much higher — Brian Goetz's classic formula is N = cores × (1 + wait/compute). A worker that waits 90 ms per 10 ms of compute can have 10× cores in the pool before saturating.

An unbounded queue is the most common production landmine. It quietly absorbs load — until you OOM, or your tail latency stretches into hours because the queue accumulated minutes of work. Bound the queue, and pick a rejection policy.

A single shared queue creates contention as N grows: every dispatch touches the same lock or atomic. Work-stealing pools (Java's ForkJoin, Go's runtime, Rust's Tokio multi-thread) give each worker its own deque. Workers push and pop their own end (no contention); when one runs dry, it steals from a busy neighbour's tail.

The shape works because most tasks come from local recursion (a worker submitting a child of its own work). The shared-queue case becomes the rare path. ForkJoinPool, Go's scheduler, and Rust's Tokio all use this trick to scale to dozens of cores without lock contention dominating.

The classic Tomcat-style server gives each request a thread, blocking on I/O. Fine up to a few thousand concurrent requests; beyond that the per-thread stack (1 MB default) and context-switching cost dominate. To go higher you either move to async I/O (one thread serves many connections via an event loop on epoll/kqueue) or to user-space threads.

Java 21's virtual threads, Go's goroutines, and Kotlin's coroutines all sit between these worlds: code looks blocking, but the runtime parks the lightweight task while a small pool of OS threads carries the work. Same dispatch model — different unit of work.

Active workers tells you how saturated the pool is. Steady-state at N is bad — there is no headroom for spikes. Queue depth tells you whether you are absorbing or shedding. Above zero for any sustained interval is back-pressure forming. Rejection rate is the canary; if your rejection policy is "abort", every spike pages someone.

Add p99 latency for the dispatch step itself (queue time, not just task time) and you have the full picture. The Little's Law identity — concurrency = arrival rate × latency — falls out of these metrics directly.

Java ExecutorService

Doug Lea's ThreadPoolExecutor (j.u.concurrent, 2004) is the canonical reference. Configurable core/max threads, work queue, rejection policy. Executors.newFixedThreadPool(N) ships with an unbounded queue — a famously dangerous default that masks back-pressure until the JVM OOMs. Production: configure explicitly with new ThreadPoolExecutor and a bounded queue.

Java ForkJoinPool / virtual threads

ForkJoinPool (Java 7) does work-stealing. Java 21's virtual threads (Project Loom) make a thread-per-request model viable again — millions of cheap virtual threads on a small carrier-thread pool, removing the historical reason to size a thread pool tightly.

.NET ThreadPool

Auto-scales from Environment.ProcessorCount up to a high ceiling (~32k by default). The CLR injects threads at ~500ms intervals when the queue is non-empty — the famous "ThreadPool starvation" stall. Modern .NET prefers Task.Run + async to manual pool sizing.

Rust Tokio runtime

Multi-threaded by default, work-stealing across N worker threads (defaults to NumCPU). Each worker has a local queue (256 entries) plus a shared injection queue. The numbers in Tokio's blog posts (2020-2024) are the benchmark anyone designing a worker pool should read.

Go runtime (GMP)

Not a thread pool exactly — goroutines are user-mode tasks scheduled onto OS threads via the GMP model (see the goroutine scheduler simulator). Effectively a thread pool of GOMAXPROCS workers, each draining a per-P run queue with work stealing.

Python concurrent.futures

ThreadPoolExecutor(max_workers=N). Defaults to min(32, os.cpu_count() + 4). The GIL means CPU-bound code doesn't benefit from threads — for that, use ProcessPoolExecutor instead.

Unbounded queue → OOM (the most common). Hundreds of public Java services have died because Executors.newFixedThreadPool(N) defaults to an unbounded LinkedBlockingQueue. Producer rate exceeds consumer rate, queue grows unbounded, JVM OOM-kills, restart, repeat. Postmortems from Google's old SRE book and Spotify's early years all cite versions of this.

Pool starvation by recursive submit. A pool of N threads where a task submits another task to the same pool and waits for its result. With N concurrent root tasks, all N threads are blocked waiting for sub-tasks that can never run. Solution: use a separate pool, or use ForkJoinPool which understands this case.

Slow downstream dragging the whole pool. One downstream service goes slow; threads in the pool block on downstream calls; queue depth grows; new requests get rejected even though the local CPU is idle. Solution: per-downstream bulkhead pools (Hystrix's signature pattern), or async I/O with a small thread pool.

Thread leak on rejected exceptions. Tasks that throw an exception don't always release pool resources cleanly — particularly when the rejection-handler itself blocks. Production Java stacks have shipped CVEs for exactly this. Always wrap submitted tasks in a try/finally that releases any held resource.

The unifying theme: thread pools fail by silently growing or silently blocking, not by loud explosions. The three metrics from Part 07 (active workers, queue depth, rejection rate) are exactly the early-warning signals for these four failure modes.