The problem › Inside the 400 cycles
One instruction, 32 addresses
We keep saying "400 cycles to memory." What actually happens in those 400 cycles?
When a warp issues a global memory load, 32 parallel threads execute the exact same instruction. But because each thread computes its own memory address, 32 independent addresses emerge from the arithmetic logic units. If the hardware blindly sent 32 separate requests to main memory every time a warp issued a load, the memory bus would instantly choke.
Instead, the addresses hit a coalescing phase. The hardware evaluates the 32 requested addresses and merges them into the minimum possible number of memory transactions. (While the programming model relies heavily on this "coalescer", the NVIDIA H100 whitepaper does not map it as a distinct physical hardware block; its functional reality is grounded in microbenchmarks rather than official die topography.)
The mechanism › Pricing the coalesce
128 bytes asked for; 1,024 bytes paid
Predict first
Same warp, same load instruction — once with contiguous addresses, once with random ones. Bytes over the bus?
Let us price the physical cost of this phase. Thirty-two lanes each requesting a 4-byte integer means the warp is asking for exactly 128 bytes of useful data. If the threads request contiguous memory locations—say, thread 0 wants byte 0, thread 1 wants byte 4, and so on—the hardware collapses the entire warp's request into a single 128-byte line: four 32-byte sectors, the minimum the transfer machinery can express. You get exactly the 128 bytes you asked for.
Compute the number › the 8× waste, derived
| useful bytes | 32 lanes × 4 B = 128 B |
| scattered: transactions | up to 32 — the coalescer can't merge strangers |
| each drags a sector | 32 B |
| bytes moved | 32 × 32 = 1,024 B |
| Result | 1,024 ÷ 128 = 8× the bandwidth for the same answer |
sector size — the 32 B sector comes from microbenchmarks, not the whitepaper.
This is the data traversal lesson from Topic 0.6, now enforced in silicon and paid per-warp rather than per-loop. Same instruction, same lanes, 8 times the traffic.
The mechanism › Down the tiers
L1, sectored L2, and the same old DRAM
Those transactions then drop into the SM's L1 data cache, a partition of the 256 KB unified memory block. If the data is missing, the request travels out of the SM to the shared L2 cache. The L2 cache is sectored. When a transaction misses in L2, the cache does not pull an entire monolithic cache line from memory; instead, the transfer unit is a smaller sector. (The H100 whitepaper omits L2 sector sizes entirely. We rely on microbenchmarks and generic architecture models to confirm that sectoring exists to prevent wasting bandwidth on unrequested bytes.)
If the data is missing from the L2 sectors, the request finally reaches the High-Bandwidth Memory (HBM). This is the payoff. HBM is not an exotic new physics paradigm invented by the GPU. It is the exact same dynamic random-access memory (DRAM) we saw in Topic 0.2. (The H100 whitepaper does not detail HBM's internal bank and row layout. We are importing Topic 0's generic DRAM model because the underlying physics are identical.)
Inside the HBM stack, the request must open a bank. A row is activated, loading its contents into the sense amplifiers, which act as a row buffer. Once the row is active, the requested data is streamed out in a burst. Everything about row-buffer locality remains perfectly true here. If those 32 threads scattered their reads across different rows, the memory controller must pay the grueling latency to precharge and activate each new row separately. If they requested contiguous data, a single row activation serves the entire burst. The GPU simply bought a massively wider bus to hide the DRAM's inherent slowness.
ChipDiagram · Level 3 · Interactive
The load tracer — the memory path
transactions
—
cycles
0
bytes moved
0/128 useful
cycles
Volta · measured
0
1,000
L2 ≈193
global
391–405
LD/ST · 32 lane addresses [WP]
coalescer — merges 32 addresses into transactions
not a WP blockL1 data cache — a partition of the 256 KB unified block [WP]
22.3 ns · measured Hoppershared L2 · 50 MB [WP] · transfer unit = 32 B sectors
159 ns · measured Hoppersector size10 × 512-bit memory controllers = 5,120-bit bus [WP]
HBM3 · 80 GB · 5 stacks [WP]· banks / rows / row buffers = Topic 0.2's DRAM
294 ns · measured Hopperinternal layoutruler = cycles, Volta-measured (Citadel §5.3: L2 ≈193 cy, global 391–405 cy) · tier costs are illustrative (30+160+210 ≈ the ~400-cycle Volta working figure; each extra row activation +100cy; random's cascade runs off the ruler) · tier notes = measured Hopper ns (Chips & Cheese) — different generation, different units, labeled · bandwidth 3,352 GB/s [WP Table 3] · ◇= not in the whitepaper · MSHRs omitted (see "what I simplified")
The consequence › Not a constant
400 cycles is a working figure, not a toll
This means the "400-cycle latency" we budget for is not a physical constant. In fact, it is a round working figure. Citadel measured 391–405 cycles of latency on the older Volta architecture, while Chips & Cheese measured roughly 294 nanoseconds of latency on Hopper.These are different chips and different units, and the "400 cycles" is simply a useful baseline for our resource math, not a strict Hopper measurement. But regardless of the generation, a fully coalesced load that hits an open HBM row buffer returns much faster than 32 scattered requests that trigger cache thrashing and dozens of distinct DRAM row activations. The latency heavily depends on what the 32 lanes asked for.
The full story › the 0.2 callback — what survives, what scales
(Mutlu's paper is a link stub in our source directory; this callback draws exclusively on PMPP Chapter 6's DRAM burst and channel models.)
What survives unchanged from the CPU DRAM story: the fundamental physics of memory access. Data still lives in capacitive cells organized into banks and rows. Before data can be read, a row must be activated and moved into sense amplifiers (the row buffer). Changing rows still incurs the strict latency penalties of precharging and activating.
What is different: the sheer physical scale. Instead of a narrow 64-bit DDR bus, an H100 uses 5 stacked HBM dies driven by 10 memory controllers, forming a 5,120-bit wide interface. Because this bus is so wide, the burst duration is exceptionally short—a massive chunk of the row buffer is transferred in a single clock cycle, enabling over 3 TB/s of bandwidth (3,352 GB/s spec-sheet, Table 3).
The full story › what I simplified: MSHRs
I deliberately omitted the presence of Miss Status Holding Registers (MSHRs) and pending request tables. When an L1 or L2 cache misses, it must allocate tracking resources to keep the pipeline moving while it waits for memory. If a kernel triggers too many scattered, uncoalesced accesses, it will completely exhaust these tracking resources, causing structural pipeline stalls before the memory bandwidth is even saturated. This simplification is honest because we are focused purely on the spatial cost of a load and its traversal down the physical memory tiers.
That was one SM's view. An H100 has 132 of them sharing L2 and HBM — who decides where blocks run and how the chip is fed?
A memory access doesn't cost 400 cycles; it costs what its path costs — and the 32 lanes chose the path. Coalesced addresses buy one 128-byte line and one open row; scattered addresses buy 8× the traffic on the same old DRAM physics from 0.2.