Inside the SM

On a CPU, switching execution from one thread to another is a violent physical event. The processor has a tiny set of execution states held in a handful of hardware registers. When the operating system preempts a thread to let another run — say, due to a timer interrupt or a blocking system call — it must perform a context switch. The processor is forced to interrupt the core, dump the outgoing thread's register contents down the memory hierarchy, and haul the incoming thread's state back up from memory. It is an operation that takes microseconds — thousands of wasted clock cycles where the processor is doing nothing but moving luggage.

Now look at a GPU. The architecture claims its Streaming Multiprocessors (SMs) can switch between executing 32-thread groups called warps every single clock cycle. If a GPU attempted to dump state to memory at that speed, the memory bus would vaporize instantly. How does the GPU switch context for free?

Walk inside the SM to find the solution isn't a clever scheduling algorithm; it is brute-force real estate.

If you are used to CPU architectures, the scale of a GPU's register storage is difficult to process. On an NVIDIA H100 SM, the register file contains 65,536 32-bit registers. That equates to exactly 256 KB of purely register storage per SM.¹

To put that in perspective, look at the SM's cache. The H100 features a combined L1 Data Cache and Shared Memory block that is also 256 KB.² But because the hardware allows developers to configure up to 228KB of that block as user-managed shared memory, the L1 proper is often reduced to a tiny 28 KB slice. This means the SM's register file is not just equal to its cache — it is an order of magnitude larger than its L1. (Both are spec-sheet numbers from the whitepaper.)

This massive architectural investment is the physical mechanism that makes zero-overhead scheduling possible. When a thread block is assigned to an SM, the hardware permanently allocates the required registers for the entire lifetime of those warps. The execution state never leaves the execution units.

SourcesPMPP 4e Ch. 4

When a warp issues a floating-point multiply-add — which on Volta hardware we measure as having a 4-cycle dependent issue latency³— or a global memory load that will take hundreds of cycles to return, the warp scheduler does not "save" the warp's state. It simply leaves the data sitting exactly where it is in the register file. The scheduler then evaluates the other resident warps, selects one that is ready to execute, and points the instruction dispatcher at it.

Because the incoming warp's registers are already physically present in the silicon, switching execution from one warp to another is not a context switch in the traditional sense. There is no save. There is no restore. The scheduler is just pointing a laser at a different row in the register array. The hardware executes the instructions for the newly selected warp on the very next clock cycle.

The consequences of this design choice completely invert how we must think about writing software. On a CPU, your primary goal is to avoid stalling. You rely on massive caches and complex out-of-order execution logic because if the CPU halts for a cache miss, the pipeline freezes. The only way a CPU can do something else while waiting for memory is to trigger that catastrophic, microsecond-long OS context switch.

On a GPU, the architecture expects you to stall. It embraces the stall. A GPU dedicates the vast majority of its silicon not to control logic or giant caches, but to holding the state of dozens of warps simultaneously. By keeping all of those contexts live in that massive register file, the SM hides the latency of memory accesses or deep arithmetic pipelines by interleaving the execution of different warps instruction by instruction. If Warp A is waiting four cycles for a math operation to finish, the SM simply fires an instruction from Warp B on cycle 2, Warp C on cycle 3, Warp D on cycle 4, and returns to Warp A on cycle 5. The latency is masked completely — for this stall.

That interleave is the whole mechanism. Watch it happen, cycle by cycle.

Now switch the instruction mix to memory-heavy. The same four warps that covered a 4-cycle math dependency perfectly cover a 400-cycle memory stall not at all — the SM sits dark for roughly 396 of every 400 cycles. Free switching bought you nothing, because there was nothing to switch to.

But this heavy reliance on physical register space creates a new, rigid hardware contract. An SM will only accept a new thread block if it has enough raw register space to physically accommodate every single thread in that block at once. If your kernel demands too many registers per thread, the SM simply refuses to load as many warps.

This brings us to a critical tension. Pointing to a different warp for free only helps if a ready warp actually exists. If a global memory load takes 400 cycles to complete, the SM needs enough alternative warps sitting in the register file to keep the execution units fed for every one of those 400 cycles. How many warps does it take to hide a massive memory stall — and given that finite 256 KB register file, what caps how many we can actually fit inside the SM?

A context switch moves state; a warp switch moves a pointer. The GPU bought that trick with 256 KB of registers per SM — and the price is a hard cap on how many warps fit.