The problem › A marvel of desperation
Billions of transistors, and the wall still stands
The modern central processing unit is a marvel of desperation. Over the last few decades, we have thrown an unbelievable amount of silicon at the problem of memory latency. We built branch predictors that attempt to guess the future. We built out-of-order schedulers that tear instruction streams apart to find independent work. We dedicated vast swaths of the silicon die to massive L2 and L3 caches, hoarding megabytes of data on the desperate hope that we won't have to go back to main memory. All of this machinery exists to hide a brutal physical reality: fetching a piece of data from off-chip DRAM takes hundreds of clock cycles.
And yet, if your workload needs to scan through a gigabyte of data, all of that brilliant machinery grinds to a halt. The caches miss, the out-of-order window fills up with stalled instructions, and the processor freezes. We spend billions of transistors trying to make a single sequence of instructions wait less, but the memory wall is still there, undefeated. So, if throwing half a silicon die at caching and reordering cannot hide a 400-cycle stall, is there another way to live with it?
Walk away from the CPU and look at the die shot of a Graphics Processing Unit (GPU). The answer is that the GPU refuses to fight the wall.
The mechanism › The transistor budget
Two ways to spend the same silicon
It comes down to a strict transistor budget. A processor's die area is finite real estate, and every square millimeter dedicated to a reorder buffer or a branch predictor is a millimeter not dedicated to an arithmetic logic unit (ALU). CPUs are latency-oriented designs. They spend their transistor budget on sophisticated control logic and massive on-chip caches to minimize the execution time of a single instruction sequence. They care deeply about how long that specific sequence has to wait.
GPUs are throughput-oriented designs. Instead of spending silicon to make one sequence wait less, they spend it on having another sequence ready to execute. If an instruction asks for a piece of data from main memory and has to wait 400 cycles, the GPU does not care. It simply puts that work to sleep and instantly switches to another independent sequence, then another, and another.
Predict first
Roughly what does halving the latency of an arithmetic operation cost in hardware?
To do this, the GPU dedicates its transistor budget not to elaborate control logic or giant caches, but to an overwhelming number of ALUs and the raw physical capacity to hold thousands of execution states at once. The memory wall has not moved, but by interleaving the execution of thousands of independent sequences, the GPU hides the latency — provided there is enough parallel work to keep the execution units busy.
The full story › Rigor check: this idea is 50 years old
While the specific terminology and massive scale of GPUs feel novel, the foundational idea of leveraging data-level parallelism through large register files and pipelined execution to hide memory latency descends directly from vector architectures that have existed for over four decades (Computer Architecture: A Quantitative Approach, Chapter 4). GPUs are descendants of a long lineage, not magic.
Interactive
The dependency chain
cycle
0
tiles done
0/64
ALUs lit 0%
1 tick ≈ 40 cycles · DRAM = 400 cycles · CPU fetch ≈ 40 cycles (caches) · each tile stands in for millions of elements
The consequence › The gamble
When the gamble fails, it fails catastrophically
But this design is a strict hardware contract, and it is a massive gamble. The throughput-oriented architecture bets its life on the existence of massive amounts of independent work. Because the GPU has stripped out the sophisticated out-of-order execution and the giant caches, its single-thread performance is agonizingly slow.
The failure fundamentally comes from insufficient parallelism. If you give a GPU a task that lacks thousands of independent work items — like traversing a linked list, or computing a running sum where step N absolutely depends on the result of step N−1 — the illusion shatters. The GPU cannot switch to another sequence because no other independent sequences exist. The single active sequence hits the memory wall and stalls for 400 cycles. Because the hardware lacks the CPU's complex latency-hiding tricks, the sequence simply halts, and the thousands of ALUs sitting on the die sit entirely dark and idle.
Compute the number › the budget, in numbers
| NVIDIA H100 (Hopper) transistors | 80 billion on 814 mm² |
| NVIDIA A100 (Ampere) transistors | 54.2 billion on 826 mm² |
| H100 L2 cache | 50 MB — larger than many CPU caches |
| die-area split (PMPP Fig. 1.1) | CPU: mostly cache + control · GPU: mostly ALUs |
| Result | philosophies, not absolutes — even the throughput chip spends 50 MB on cache |
SourcesHopper In-Depth pp. 2–3Ampere whitepaper Key features
The full story › What I simplified
I simplified the dichotomy between the two architectures. By setting up a strict binary between "CPU = caches/control" and "GPU = ALUs", I deliberately ignored the fact that modern CPUs feature extremely wide vector math units to improve their throughput. I also ignored that modern GPUs actually dedicate substantial silicon to caching — as noted in the numbers above, the H100 features a 50 MB L2 cache, which is larger than many CPU caches.
This simplification is honest for an opening chapter because we are establishing the foundational design philosophies. While the lines have blurred over the last decade, the core architectural trade-off — spending transistors on single-thread control versus parallel arithmetic throughput — remains the defining split between how the two processors are built to operate.
This brings us to a fundamental mechanical problem. To successfully tolerate these long memory latencies, a modern GPU Streaming Multiprocessor might have 2,048 independent threads of executionassigned to it at the exact same time. On a traditional CPU, every active thread requires its own instruction fetch and decode hardware — a "front-end" to read the instruction stream. But replicating a fetch-and-decode front-end 2,048 times would instantly bankrupt our silicon budget, eating all the space we just saved for our ALUs. How does the hardware manage 2,048 active threads without paying for 2,048 instruction fetchers?
CPUs spend silicon so one thread waits less; GPUs spend it so waiting doesn't matter. The bet only pays when you bring thousands of independent work items — and it fails catastrophically when you don't.