The problem › The 400-cycle gap
How many warps does it take?
When your kernel executes a global memory load, the hardware sends a request down the memory hierarchy. Because that data resides in off-chip memory, it will take approximately 400 clock cycles to return. During that 400-cycle window, the warp that issued the load is paralyzed. To prevent the Streaming Multiprocessor (SM) from sitting dark, the warp scheduler must instantly pivot to another warp and issue its next instruction.
But how many alternate warps do you actually need to cover the entire 400-cycle gap? The number of warps you need is not a programming preference; it is a mathematical mandate. It is governed by Little's Law, which states that the parallelism required to keep a pipeline busy is exactly equal to the pipeline's latency multiplied by its throughput: Instructions in Flight = Latency × Throughput.
The mechanism › Little's Law
1,600 instructions in flight
Let's do the arithmetic together. Your memory pipeline has a latency of roughly 400 cycles. We know from the hardware spec that an SM contains 4 warp schedulers, each capable of issuing one instruction per clock cycle. That gives the SM a peak throughput of 4 instructions per cycle.
Compute the number › Little's Law prices the stall
| memory latency | ≈ 400 cycles (measured, Volta) |
| SM throughput | 4 schedulers × 1 instr/cycle = 4 instr/cycle |
| in flight = latency × throughput | 400 × 4 = 1,600 instructions |
| if each stalled warp contributes one | you'd need 1,600 resident warps |
| the hardware has | 64 warp slots |
| Result | even one fully-fed scheduler needs 400 in flight — hence the massive 64-slot file |
To maintain 1,600 instructions in flight, how many warps do we need? If we assume that each warp only contributes one outstanding instruction at a time—which is typically true when a warp is blocked on a dependent memory load—this math suggests you would need an impossible 1,600 resident warps. The hardware doesn't have 1,600 warp slots; it has 64. But even keeping just a single one of those four schedulers fully fed requires 400 instructions in flight (400 cycles × 1 instruction/cycle). Simply covering a 400-cycle latency requires hundreds of instructions in flight per SM, which is exactly why the hardware provides a massive 64 warp slots to begin with.
The mechanism › Three budgets
Warps do not fit onto the SM for free
You cannot simply launch your kernel and assume the hardware will fill all 64 slots. The SM has three strict hardware budgets, and hitting any one of them places a hard cap on your resident warps: warp slots (exactly 64 resident warps), shared memory (a finite block of SRAM, up to 228 KB, partitioned among your thread blocks), and the register file (exactly 65,536 32-bit registers — 256 KB).
The register file is almost always the binding constraint, because registers-per-thread is the budget you directly control. Let's compute an occupancy cap.
Compute the number › the register cap
| the physical maximum | 64 warps × 32 threads = 2,048 threads |
| registers each may claim | 65,536 ÷ 2,048 = 32 regs/thread |
| but your kernel wants 64 regs | 65,536 ÷ (64 × 32) = 32 warps |
| Result | the SM refuses to load more — occupancy halves, and the 400 cycles peek through |
Interactive
The occupancy calculator
1 · TLP — two budgets cap your warps
Registers per thread · 32
Shared memory per block · none
Threads per block
Resident warps
64/64
occupancy 100%
65,536 ÷ (32 × 32) = 64 warps
warp slots · 4 blocks × 16 warps
2 · ILP — spend registers on independent work instead
Independent loads per thread (ILP) · 1 → 40 regs/thread
Est. kernel time
16,400 cy
baseline
warp slots · occupancy 80%
H100 spec-sheet: 64 warp slots · 4 schedulers · 65,536 regs · shared ≤ 228 KB (whitepaper Table 4 / Fig. 7) · latency ≈ 400 cy (measured 391–405, Volta, Citadel §5.3) · panel 1 allocates block-granular (the real allocator); panel 2 uses the prose's warp-granular simplification · ILP register cost (24 + 16/chain), workload (2,048warp-loads) and wave model are illustrative — the spec's "utilization pinned at 100%" is shown as loads-in-flight rising while time falls
The consequence › Volkov's correction
Maximum occupancy is not the goal
The received wisdom in GPU programming dictates that you must restructure your code to use fewer registers, forcing the hardware to accept more warps and push occupancy back to 100%. This intuition is wrong.
Predict first
Kernel A: 100% occupancy, each warp one outstanding load. Kernel B: 20% occupancy, but each thread keeps 8 independent loads in flight. Which hides the 400-cycle latency better?
As Vasily Volkov demonstrated, maximum occupancy is not the goal.Thread-Level Parallelism (TLP)—swapping between different warps—is not the only way to hide latency. Instruction-Level Parallelism (ILP) within a single warp hides latency just as effectively. Remember the assumption we made when calculating our Little's Law requirement? We assumed each warp could only have one outstanding instruction at a time. This is exactly the assumption that ILP breaks. If you spend your register budget to keep more independent variables alive within a single thread, you can compute multiple independent outputs at once. If one thread issues four independent math instructions before finally stalling on a memory dependency, the hardware seamlessly overlaps those independent instructions to fill the idle cycles. Independent instructions in one thread cover a stall exactly the same way instructions from an entirely different warp do.
Therefore, deliberately spending your registers to maximize ILP per thread is often vastly superior to sacrificing registers to inflate your warp count. Doing more work per thread means you do not have to spill intermediate data to slow memory. Your occupancy number will plummet—perhaps down to a fraction of the maximum—but your actual execution performance will rise, provided there is enough independent work per thread to saturate the pipeline.
The full story › What I simplified: block-level quantization
When calculating the register cap in the prose above, I divided the total register file by raw threads (65,536 ÷ 2,048) to find the threshold of 32 registers. This implies that warps are evicted from the SM one by one. In reality, the SM allocates resources at the thread block level, not the warp level. If your thread block contains 8 warps, the SM must have enough free registers to fit all 8 warps at once; otherwise, it rejects the entire block in a single chunk. (Panel 1 of the interactive above allocates block-granularly — that is why the warp count falls in harsh steps.)
This simplification is honest because introducing block-level quantization math here distracts from the core conceptual mechanism of Little's Law. The fundamental physical reality remains unchanged: the raw division of finite register real estate dictates exactly how much parallelism the SM can hold.
If ILP and TLP are both just mechanisms to keep the execution units firing while we wait for data... we keep saying "400 cycles to memory." What actually happens in those 400 cycles?
Little's Law prices the stall: latency × throughput = 1,600 instructions in flight. Occupancy buys them with warps; ILP buys them with registers — and the product, not the occupancy percentage, is what hides the 400 cycles.