The problem › Naming your discovery
You already did this in 0.X
We know the cost of every path in the machine. Can we predict a kernel's runtime before writing it?
In your Topic 0 capstone, you actually already did this. You took the peak compute of a processor, divided it by its peak memory bandwidth, and produced a bare operations-per-byte ratio. Then you looked at the loops you wrote, counted their operations against their memory accesses, and realized they all fell woefully short of the machine's ratio. You discovered the boundary between compute-bound and memory-bound code. Now, we give your discovery its formal name.
The ratio you calculated for your kernel is its arithmetic intensity: the total number of floating-point operations (FLOPs) performed divided by the total number of bytes transferred from main memory. The ratio you calculated for the hardware is its ridge point: the machine's peak floating-point performance divided by its peak memory bandwidth.
If we plot a machine's capabilities on a log-log graph, with arithmetic intensity on the X-axis and achievable performance (FLOP/s) on the Y-axis, we construct the Roofline model. The plot features two intersecting ceilings: a slanted roof rising from the origin, representing the absolute limit of memory bandwidth, and a flat, horizontal roof representing the hardware's peak computational limit. The ridge point is exactly where they meet. Every kernel you write exists as a single point beneath this structure. To the left of the ridge you hit the slanted roof: strictly memory-bound, runtime dictated entirely by bytes ÷ bandwidth. To the right, you hit the flat roof: compute-bound.
This picture completely reorients how we optimize software. "Make it faster" is the wrong question. "Which ceiling am I under?" is the right one. On the slanted slope, optimizing your math is entirely wasted effort; only moving fewer bytes will help. On the flat roof, optimizing memory access gains you nothing; you must find more FLOPs to issue per cycle.
The mechanism › Three kernels under the roof
vecadd, GEMV, and the N/6 revelation
Compute the number › arithmetic intensity, derived
| vecadd · x[i] = y[i] + z[i] | 1 FLOP ÷ (8 B read + 4 B write) = ~0.08 FLOPs/B |
| GEMV · y = Ax + y | 2N² FLOPs ÷ 4N² bytes = 0.5 FLOPs/B — constant, N cancels |
| GEMM · C = AB | 2N³ FLOPs ÷ 12N² bytes = N/6 — grows with N |
| Result | GEMM is one of the only kernels that can scale itself out from under the memory roof |
This should land as a revelation. For vector addition and GEMV, the arithmetic intensity is a flat, unchangeable constant. But for GEMM, the arithmetic intensity grows with N. The FLOPs scale as N³ while the memory footprint only scales as N². This is exactly why matrix multiplication is the foundational operation of modern AI, and why GPUs were physically built to execute it: it is one of the only operations that can mathematically be scaled far enough to the right to escape the slanted memory roof.
Interactive
The roofline plot
Compute roof · precision [WP · Table 1·?]
arithmetic intensity
682 FLOP/B
achievable
67 TFLOP/s
est. runtime
2.05 ms
Bytes moved · 201 MB
FLOPs computed · 137 GFLOP
peaks [WP Table 1]: FP64 33.5 · FP32 66.9 · FP16 TC 989.4 TFLOPS · bandwidth 3,352 GB/s [WP Table 3] — peak FLOP/s is Level 2's lanes and Tensor Cores; bandwidth is Level 5's HBM pipe · overhead ceiling ~2µs is illustrative (Horace He's regime) · intensities assume compulsory traffic (see "what I simplified")
The consequence › The moving ridge
The ridge point depends on which math you buy
Predict first
GEMM at N=1,024 has arithmetic intensity ≈ 171 FLOPs/B. On H100 FP16 Tensor Cores (ridge ≈ 295), which ceiling is it under?
Compute the number › H100 ridge points (peak ÷ bandwidth)
| FP64 | 33.5 TFLOPS ÷ 3,352 GB/s = 10.0 FLOPs/B |
| FP32 | 66.9 TFLOPS ÷ 3,352 GB/s = 20.0 FLOPs/B |
| FP16 Tensor Core | 989.4 TFLOPS ÷ 3,352 GB/s = 295.2 FLOPs/B |
| Result | switch on the Tensor Cores and almost everything becomes memory-bound |
However, the classic Roofline model is incomplete. It assumes your workload is large enough to saturate the hardware. If you launch a tiny workload—say, multiplying two 4×4 matrices—you will not hit the compute roof, nor will you hit the memory roof. You will hit a third regime: overhead-bound. The time spent in the Python interpreter, the deep learning framework, and the PCIe launch latency will dwarf the actual execution time. The GPU will sit idle waiting for the CPU to tell it what to do. The classic Roofline model does not draw this ceiling, but in practice, it is often the first one you hit.
The full story › what I simplified: compulsory traffic
When calculating the arithmetic intensity of GEMV and GEMM above, I only accounted for the compulsory memory traffic—the absolute minimum number of bytes that must be read from main memory to satisfy the math. In reality, unless a kernel is perfectly tiled to leverage the L1/L2 caches and the SM shared memory, the hardware will inevitably fetch the same data multiple times, artificially inflating the bytes moved and driving the actual arithmetic intensity even further to the left on the roofline plot. This simplification is honest because the Roofline model is fundamentally an upper-bound analysis: compulsory traffic establishes the highest possible operational intensity the algorithm can achieve before caching inefficiencies drag it backward.
You can now read a spec sheet like a story. Prove it.
Arithmetic intensity names what you derived in 0.X: FLOPs ÷ bytes against the machine's ridge. Left of the ridge, only moving fewer bytes helps; right of it, only more math helps — and the Tensor Cores move the ridge so far right that almost everything lands back under the memory roof.