The problem › Zooming out
One SM was the easy part
We just traced the life of a single memory load from a single warp inside a single Streaming Multiprocessor. That was one SM — who schedules blocks across 132 of them, and how is the whole chip fed?
To understand the machine at scale, we have to zoom out. The SMs do not float in a void; they are strictly organized. Two SMs are grouped together into a Texture Processing Cluster (TPC), and those TPCs are bundled into larger Graphics Processing Clusters (GPCs). The entire die is bisected by a massive 50 MB L2 cache, while 5 HBM3 memory stacks sit at the physical periphery of the silicon, feeding the compute cores through ten 512-bit memory controllers.
The mechanism › Binning
The die they designed vs the part you bought
But if you look closely at a photograph of the bare silicon, you will find a discrepancy. The physical GH100 die actually contains 144 SMs, 72 TPCs, 60 MB of L2 cache, and 6 HBM sites. Yet the H100 SXM5 part you buy and program only gives you access to 132 SMs, 66 TPCs, 50 MB of L2, and 5 HBM stacks.
Compute the number › the binning delta
| TPCs | 72 designed → 66 shipped |
| SMs | 144 → 132 |
| L2 | 60 MB → 50 MB |
| HBM stacks | 6 → 5 |
| memory controllers | 12 → 10 (512-bit each) |
| Result | yield and binning — the spec sheet tells the hard truth about what you own |
This gap between the die they designed and the part you bought is due to the reality of manufacturing. Silicon is defective; fabricating a chip with 80 billion transistors flawlessly every single time is physically impossible. Rather than throwing away chips with a few flawed SMs or a bad memory controller, the manufacturer permanently disables the defective sections and bins the good remaining dies into shipping products. (The whitepaper gives the total count of 66 active TPCs across 8 GPCs, but not the per-GPC split; we cannot assert exactly how the missing 6 TPCs are distributed.)
The mechanism › Transparent scalability
You don't choose where blocks run
So, you have 132 SMs. When you write a program, you launch a grid of thread blocks. But you do not choose which block runs on which SM. The hardware scheduler takes your grid and assigns blocks to whichever SM currently has available room.
This hardware-controlled scheduling is what provides transparent scalability. Because you do not hardcode blocks to specific SMs, the exact same compiled binary will run correctly on a laptop GPU with 20 SMs and a data-center H100 with 132 SMs. But this scalability enforces a brutal restriction: blocks cannot communicate with each other, nor can they wait for each other to finish. Because the scheduler assigns them dynamically, Block 0 and Block 131 might run at the exact same time on opposite sides of the chip, or they might run years apart, in any order. The restriction—that blocks must be completely independent—is the exact mechanism that makes the scalability possible.
ChipDiagram · Level 4 · Interactive
The binning and the wave — the die
waves
2
exact
1.008
wall-clock vs ideal
×1.98
GPC 0
GPC 1
GPC 2
GPC 3
GPC 4
GPC 5
GPC 6
GPC 7
8 GPCs × 9 TPCs × 2 SMs = 144 designed [WP] · SXM5 ships 132 [WP] · grey = binned-off silicon
Grid size · 133 blocks (1 block/SM assumed)
die counts [WP], H100 whitepaper "Architecture In-Depth" · dead-TPC arrangement is ◇ conceptual (total published, split not) · 1 tick = 1 wave · waves = ⌈grid ÷ 132⌉, wall-clock ×1.98= waves ÷ exact — computed, not asserted · 1 block/SM is 1.4's capacity simplification (see "what I simplified")
The consequence › Wave quantization
133 blocks cost twice what 132 do
Predict first
Your kernel fits 1 block per SM. You launch 133 blocks on 132 SMs instead of 132. Wall-clock cost?
Because you do not control the scheduling, the math of how your blocks divide into the SM count dictates your performance. Imagine you launch a grid of 133 blocks, and assume your kernel demands so many registers that each SM can physically only fit one block at a time. The hardware scheduler launches the first 132 blocks across the 132 SMs — your first “wave.” The 133rd block must wait until an SM finishes and frees up resources. When the first wave completes, the scheduler launches the second wave—consisting of exactly one block. The other 131 SMs sit dark and idle. This is wave quantization, and it bites hardest when your grid size is just barely larger than a multiple of the chip's capacity.
Compute the number › the wave arithmetic
| hardware | 132 SMs · 1 block/SM assumed |
| small grid | 133 ÷ 132 = 1.007 → ⌈⌉ = 2 waves ≈ 2× wall-clock |
| large grid | 1,321 ÷ 132 = 10.007 → 11 waves ≈ +10% |
| Result | wave quantization is a small-grid pathology — the tail amortizes as the grid grows |
The full story › what I simplified: 1 block per SM
For the wave arithmetic, I assumed that each SM could physically hold exactly one block. In reality, as we saw in the resource math of Chapter 1.4, an SM can hold multiple blocks concurrently if the register and shared memory budgets permit it. This simplification is honest here because wave quantization happens at the capacity limit, whatever that multiple happens to be. If your kernel allows 2 blocks per SM, your chip capacity is 264 blocks — and a 265-block grid triggers the exact same catastrophic second wave. The math scales, but the penalty mechanism remains identical.
The data all this machinery consumes sits in HBM — how did it get there, and what does the trip cost?
You program the binned part, not the designed die — and you don't choose where blocks land. The scheduler fills waves of 132; grids that spill one block past a wave boundary pay for a whole extra wave.