Computer Architecture · Primer 03

The Memory Hierarchy

Why a modern processor can execute billions of instructions per second yet still spend most of its time waiting for data — and how six tiers of storage, each a thousandfold faster than the last, paper over the gap.

By Marisol Tanaka  ·  Reviewed by the Systems Reading Group

A CPU in 2026 can issue an arithmetic operation in roughly a quarter of a nanosecond. Fetching that operation's operand from main memory, though, can take one hundred nanoseconds or more — a four-hundredfold disparity. If every instruction had to wait on DRAM, a fast core would spend 99.75% of its time idle. The fact that it does not is the entire reason the memory hierarchy exists.

The hierarchy is a stack of progressively smaller, faster, costlier stores, each acting as a cache for the level below. The trick is that programs do not touch memory at random: they exhibit locality. Exploit that locality well and the processor almost never has to pay the full DRAM penalty.

01Two kinds of locality

Everything in the hierarchy rests on two empirical observations about how real programs behave. Neither is a law of physics; both are overwhelmingly true in practice.

Temporal locality

If a program accesses a memory location now, it is likely to access the same location again in the near future. Loop variables, stack frames, and recently-fetched instructions all get re-read within microseconds. The implication: keep recently-used data close to the core, where re-access is cheap.

Spatial locality

If a program accesses a location, it is likely to access nearby locations soon. Arrays are scanned sequentially, structs are read field-by-field, instructions stream forward. The implication: fetch data in blocks, not single words — one slow trip to DRAM can amortize across many fast cache hits.

02The six tiers, fastest first

Walk down the pyramid and each level is roughly an order of magnitude larger and an order of magnitude slower than the one above it. The numbers below are typical for a desktop-class x86 core in 2026; mobile and server parts differ in degree, not in kind.

Registers

The fastest storage of all lives inside the core itself. A modern x86-64 core exposes 16 general-purpose registers plus a vector register file; ARMv9 gives 31. Access latency is effectively zero — a register operand is available the same cycle the instruction issues. The cost is scarcity: you cannot hold more than a few dozen values at once, so the compiler spends considerable effort deciding what earns a register slot and what gets spilled to the stack.

Why registers win

Nothing else in the hierarchy is addressed by name. A register is selected by a field in the instruction word itself — there is no address translation, no tag check, no cache lookup. That directness is the entire performance advantage, and it is why the compiler's register allocator is among the most impactful passes in the toolchain.

Cache: L1, L2, L3

Beyond the registers, the hierarchy continues inside on-die SRAM. Each core has a private L1 split into separate instruction and data banks (typically 32–48 KB each, ~1 ns), and a private unified L2 (256 KB–2 MB, ~4 ns). All cores share a single L3 (16–96 MB, ~12 ns) — the last level of cache before the request must leave the chip.

Cache lines, not words

Caches do not store individual bytes. The unit of transfer is the cache line — a fixed 64-byte block on virtually every contemporary architecture. A single load of one int pulls the surrounding fifteen neighbors into L1 for free, banking on spatial locality. This is why a struct-of-arrays can outperform an array-of-structs by surprising margins, and why malloc returns 16-byte-aligned memory: the allocator is anticipating the line size.

Associativity and replacement

When a line must be evicted to make room, the cache consults a replacement policy — usually an approximation of least-recently-used. How many candidate slots a line can map to is its associativity: an 8-way set-associative L1 means each memory address has eight possible homes. Higher associativity reduces conflict misses at the cost of a slower tag lookup.

Main memory (DRAM)

Off-chip dynamic RAM is where working sets too large for cache actually live. Latency is dominated not by the data wires but by the row-activate precharge cycle of the DRAM cell array — a physical constraint of the capacitor-and-transistor storage cell. A sequential access to an already-open row is fast (~20 ns); a random access to a closed row can exceed 100 ns. This open-row asymmetry is precisely what cache lines exploit: a line fill keeps the row open long enough to stream 64 useful bytes.

The bandwidth-vs-latency split

Modern DRAM is engineered for bandwidth, not latency. A DDR5 channel can transfer tens of gigabytes per second once a transfer begins — but the first byte still pays the full row-activate tax. Systems that stream large arrays (GPUs, vector code) care about bandwidth; systems that chase pointers (databases, JavaScript engines) are murdered by latency. The hierarchy cannot fix the latency, so it tries to ensure you almost never have to pay it.

Storage: SSD and beyond

Below DRAM sits persistent storage, addressed not by load instructions but by system calls. A fast NVMe SSD reaches the controller in ~10 µs — already a hundred times slower than DRAM. A spinning hard drive adds seek time measured in milliseconds, and network-attached storage pushes into tens of milliseconds. At these latencies the hierarchy stops being a cache and starts being an archive: the working set must be lifted into DRAM before any real work can happen.

03The latency in human terms

The raw nanosecond figures are hard to feel. Dean and Barroso's classic scaling — multiply every delay by a billion — makes the hierarchy visceral:

• L1 cache hit: 1 ns  →  1 second
• L3 cache hit: 12 ns  →  12 seconds
• DRAM access: 100 ns  →  100 seconds
• SSD read: 15 µs  →  4 hours
• Disk seek: 5 ms  →  58 days
• Packet round-trip: 150 ms  →  5 years

On this scale, fetching from L1 is a single heartbeat; fetching from DRAM is under two minutes of standing still; fetching a packet from across the ocean is half a decade. The cache exists so that, almost all the time, the program never leaves the heartbeat tier.

04What this means for the code you write

The hierarchy is invisible in source but decisive in performance. A handful of habits move the needle more than any micro-optimization:

Be kind to the cache line

Contiguous access patterns let each 64-byte line do maximal work. A loop that strides through an array sequentially hits L1; the same loop that walks a linked list of separately-allocated nodes pays a potential DRAM miss per hop. Data layout is performance.

Mind the working set

Every cache level has a size cliff. Fit your hot data inside L2 and it stays warm; let it spill past L3 and every access risks a round-trip to DRAM. The boundary is not theoretical — profiler traces show a visible step function when an algorithm's footprint crosses a cache boundary.

Prefetching is real, but not magic

Hardware prefetchers watch access streams and fetch ahead, converting predictable sequential patterns into cache hits before the load is even issued. They are excellent at strides and linear scans, poor at pointer chases. Code that accesses memory in patterns a prefetcher can recognize runs dramatically faster than code that does not — for free.

05The takeaway

The memory hierarchy is not an implementation detail you work around. It is the shape of the machine, and good code is code whose access patterns rhyme with that shape.

— Systems Reading Group, margin note

Registers, three cache levels, DRAM, and persistent storage — six tiers, six orders of magnitude in latency, unified by one principle: keep the data you are about to use as close to the core as you possibly can. Write code that respects locality and the hierarchy disappears into raw speed. Write code that fights it and you will spend your budget waiting, one hundred nanoseconds at a time.

This primer is the third in a twelve-part series on computer architecture fundamentals. The next installment, Primer 04 — Pipelining & Superscalar Execution, picks up where the memory hierarchy leaves off: what happens to all those fast register operands once they reach the execution units.