Computer architecture
How the hardware runs your code: what a processor does with each instruction, why a read from memory can take a hundred times longer than an addition, and how drives, network cards and GPUs move data. It is written for programmers, and the 15 chapters use figures from chips you can buy today.
You do not need to design chips to get value from this. Most of it explains performance problems you can see from ordinary code: slow loops over large data, threads that get slower when you add more of them, and servers that behave differently on bigger machines.
The chapters
The processor
From transistors up to a core that runs several instructions per clock cycle.
- 01 Transistors, gates and the ALU
How transistors make logic gates, and how gates combine into a circuit that adds and compares numbers.
Try it: Build a CPU from NAND gates - 02 Clocks, latches and flip-flops
How a chip holds a value from one clock tick to the next, and why clock speeds stopped rising at around 5 GHz.
- 03 The instruction cycle
Fetch, decode, execute and write back, following one RISC-V instruction through the processor.
- 04 Pipelining
Overlapping instructions like an assembly line: the classic five stages, and why real cores use many more.
- 05 Branch prediction and speculation
How a CPU guesses which way a branch will go before it knows, what a wrong guess costs, and how Spectre abused the guessing.
- 06 Out-of-order execution
Running instructions as soon as their inputs are ready instead of in program order, using register renaming and a reorder buffer.
- 07 SIMD and vector instructions
One instruction working on many values at once (SSE, AVX, AVX-512, NEON and SVE), and when the compiler uses them for you.
Memory
Where data lives, how long it takes to reach, and what the hardware does to hide the wait.
- 08 The memory hierarchy
Registers, caches, RAM and storage, and the large gaps in speed between each level.
- 09 CPU caches and coherence
Cache lines, associativity, how cores keep their caches in agreement (MESI), and false sharing.
Try it: CPU cache simulator - 10 Virtual memory and the TLB
How every address goes through page tables, how the TLB keeps that fast, and when huge pages help.
Try it: Virtual memory simulator - 11 NUMA and memory bandwidth
Why memory attached to another socket is slower to reach, and how Linux decides where memory and threads go.
Devices
How data gets between the CPU, drives, network cards and GPUs.
- 12 PCIe, DMA and interrupts
How network cards and drives copy data straight into memory, and how they tell the CPU it has arrived.
- 13 SSDs and the flash translation layer
Why flash is written in small pages but erased in large blocks, what the drive’s firmware does about it, and NVMe queues.
- 14 GPUs and accelerators
Thousands of simple cores running the same code on different data, high-bandwidth memory, and chips built for machine learning.
Start-up
What happens between pressing the power button and the kernel taking over.
- 15 From power-on to the kernel
Reset, firmware (UEFI), the bootloader and the kernel, plus microcode updates, Secure Boot and the TPM.
Ideas that explain most performance problems
If you remember nothing else from this path, these are the points that come up again and again when code is slower than expected.
Most slow code is waiting for memory
A core can finish several instructions every nanosecond, but a read that has to go all the way to RAM takes around 80 to 100 ns. Code that misses the cache often spends most of its time waiting.
Read “The memory hierarchy”Caches load whole lines, not single variables
Reading one byte brings in the whole cache line around it: 64 bytes on most x86 and Arm chips, 128 on Apple’s. Data stored next to each other and read in order is cheap.
Read “CPU caches and coherence”Threads can slow each other down without sharing data
If two threads write different variables that sit in the same cache line, the line moves back and forth between their cores on every write. This is false sharing, and placing the variables on separate lines fixes it.
Read “CPU caches and coherence”Every address is translated before it is used
Programs use virtual addresses. The CPU looks up the physical address in the TLB, and a miss means reading the page tables from memory. Huge pages reduce how often that happens.
Read “Virtual memory and the TLB”Pipelining raises throughput, not the speed of one instruction
An instruction still takes many cycles from fetch to finish, but a new one can start every cycle, so a core has many in progress at once.
Read “Pipelining”The CPU guesses, and wrong guesses are expensive
Branch predictors are right most of the time on ordinary code. When one is wrong, the work done on the wrong path is thrown away, which costs roughly 15 to 20 cycles on a modern core.
Read “Branch prediction and speculation”Instructions do not run in the order you wrote them
An out-of-order core runs whichever instructions have their inputs ready, then makes the results visible in program order. Hundreds of instructions can be in flight at once.
Read “Out-of-order execution”Clock speeds stopped rising, so chips got wider
Power rises steeply with clock speed, so since the mid-2000s most gains have come from more cores, wider vector units and specialised hardware instead of faster clocks.
Read “Clocks, latches and flip-flops”
How long things take
Rough figures for a current desktop or server. The bars use a logarithmic scale, so each step to the right is about ten times slower. The cycle column assumes a 3 GHz clock.
| Operation | Time | Clock cycles | Relative time |
|---|---|---|---|
| Register | 0.3 ns | 1 | |
| L1 cache | 1 to 1.5 ns | 3 to 5 | |
| L2 cache | 3 to 5 ns | 9 to 15 | |
| L3 cache | 10 to 20 ns | 30 to 60 | |
| RAM on the same socket | 80 to 100 ns | 240 to 300 | |
| RAM on another socket | 130 to 150 ns | 390 to 450 | |
| NVMe SSD, random 4 KB read | 50 to 100 µs | 150,000 to 300,000 | |
| Round trip inside a data centre | 100 to 500 µs | 300,000 to 1.5 million | |
| Hard disk seek | 5 to 10 ms | 15 million to 30 million | |
| Round trip across an ocean | 150 ms | 450 million |
Based on Jeff Dean and Peter Norvig’s well-known list, updated for current hardware. Real numbers vary with the chip, the memory and the load on the machine.
Four current cores compared
Apple’s cores decode more instructions per cycle and run at lower clock speeds. Intel and AMD run at higher clock speeds. The SiFive U74 is a small, simple core that runs instructions in order, of the kind used in development boards and embedded systems.
| Apple M4 performance core | Intel Raptor Cove (13th and 14th gen Core) | AMD Zen 5 | SiFive U74 | |
|---|---|---|---|---|
| Instruction set | Arm | x86-64 | x86-64 | RISC-V (RV64GC) |
| Execution | Out of order | Out of order | Out of order | In order, two at a time |
| Instructions decoded per cycle | About 10 | 6 | 8 (two blocks of 4) | 2 |
| L1 data / instruction cache | 128 KB / 192 KB | 48 KB / 32 KB | 48 KB / 32 KB | 32 KB / 32 KB |
| L2 cache | 16 MB, shared by the performance cores | 2 MB per core | 1 MB per core | 2 MB, shared (StarFive JH7110) |
| L3 cache | None; a system-level cache instead | Up to 36 MB | 32 MB per 8-core chiplet | None |
| Cache line | 128 bytes | 64 bytes | 64 bytes | 64 bytes |
| Vector instructions | NEON (128-bit), SME | AVX2; AVX-512 is turned off | AVX-512 | None |
| Top clock speed | About 4.5 GHz | Up to 6.2 GHz (Core i9-14900KS) | 5.7 GHz (Ryzen 9 9950X) | 1.5 GHz (StarFive JH7110) |
Figures come from vendor specifications where vendors publish them. Apple does not publish its decode width; the figure here is from independent measurements.
Common mistakes
- Counting instructions instead of cache misses. A short sequence of instructions that misses the cache ten times is slower than a longer one that stays in L1.
- Expecting hyperthreading to double throughput. A second thread on the same core shares its execution units and caches. The gain is often in the range of 15 to 30% and can be negative for code that already keeps the vector units busy.
- Using volatile to synchronise threads. In C and C++,
volatilestops the compiler from keeping a value in a register. It does nothing about the CPU reordering memory operations. Use atomics or locks. - Forgetting that the page cache uses the same RAM. Reading a very large file pushes other data out of memory, which can slow down every other process on the machine.
- Testing on one socket and deploying on several. Memory on another socket can take up to about twice as long to reach. Profile on the hardware you deploy to.
- Padding to 64 bytes on Apple silicon. Apple’s chips use 128-byte cache lines, so 64 bytes of padding does not separate two variables. Use the line size of the machine you run on.
Books and courses
- Computer Systems: A Programmer’s Perspective, by Bryant and O’HallaronThe best single book for programmers. Used in CMU’s 15-213 course.
- Computer Organization and Design (RISC-V edition), by Patterson and HennessyThe standard undergraduate textbook, from logic gates to caches.
- Computer Architecture: A Quantitative Approach, by Hennessy and PattersonThe graduate follow-up, with more on multicore chips, coherence and memory.
- A Primer on Memory Consistency and Cache Coherence, by Nagarajan, Sorin, Hill and WoodA short book on how cores agree about memory. Free from the publisher.
- What Every Programmer Should Know About Memory, by Ulrich Drepper (2007)A long paper. The numbers are dated; the explanations of caches and NUMA still hold.
- Understanding Software Dynamics, by Richard L. SitesHow to measure where time goes in real server software.
- CoursesMIT 6.004 (now 6.191), CMU 15-213, Berkeley CS61C, and Nand2Tetris, which has you build a working computer starting from NAND gates.
- ManualsAgner Fog’s optimisation guides and instruction tables, the Intel and AMD optimisation manuals, and the Arm Architecture Reference Manual.
Tools for measuring
- perfLinux profiler and hardware counters.
perf c2cfinds cache lines that threads fight over. - InstrumentsThe macOS profiler. The CPU Counters template reads Apple silicon’s performance counters.
- VTune and uProfIntel’s and AMD’s profilers, with detail on stalls in their own cores.
- Compiler ExplorerShows the assembly a compiler produces for your code, at godbolt.org.
- llvm-mcaEstimates how many cycles a block of assembly takes on a given core.
- numactl and numastatControl and inspect which NUMA node memory and threads use on Linux.
- pmu-tools (toplev)Intel’s top-down method, which splits stalls into front-end, back-end, bad speculation and retiring.
Related paths
- Operating systemsWhat the kernel builds on top of the hardware: processes, threads, scheduling and virtual memory.
- DatabasesSoftware designed around caches and storage, from B-trees to the page cache.
- NetworkingThe other side of the network card, from packets to TCP.
- Go internalsA language runtime on top of all this: its scheduler, garbage collector and memory model.