The problem › The crossing
The bytes don't start on the GPU
The data all this machinery consumes sits in HBM — how did it get there, and what does the trip cost?
For six chapters, we have described a machine that consumes bytes at staggering speeds. But those bytes do not originate on the GPU. They begin their life in the host CPU's system memory. To reach the execution lanes, data must cross the physical boundaries between chips.
To understand the gravity of this crossing, look at the physical pipes connecting the system. An H100 SXM5 GPU features 80 GB of High-Bandwidth Memory (HBM3) that delivers 3,352 GB/s of bandwidth. If the GPU needs to talk to a peer GPU on the same motherboard, it uses the NVLink interface, which provides 900 GB/s of total bandwidth. But to talk to the host CPU, the data must travel over a PCI Express (PCIe) Gen 5 link. That link provides exactly 64 GB/s of bandwidth in each direction.
The mechanism › The ratio
A drinking straw feeding a fire hose
Compute the number › the three pipes, priced
| HBM ÷ PCIe | 3,352 ÷ 64 = 52.3× |
| NVLink ÷ PCIe | 900 ÷ 64 = 14.0× |
| consume 80 GB from HBM | 80 ÷ 3,352 ≈ 24 ms |
| refill 80 GB over NVLink | 80 ÷ 900 ≈ 89 ms |
| refill 80 GB over PCIe | 80 ÷ 64 = 1.25 s |
| Result | the machine drains itself 52× faster than the host can refill it |
The gap between these three numbers dictates the entire software architecture of GPU computing. The PCIe link to the host is roughly 1/52nd the bandwidth of the HBM. The machine can physically consume its entire 80 GB of memory in about 24 milliseconds. Refilling that same 80 GB over the PCIe bus takes 1.25 seconds. The host link is a drinking straw feeding a fire hose.
This means the fundamental strategy in GPU programming is not to "transfer faster." It is to not transfer. You must get the data resident on the device and keep it resident. Every time you cross the PCIe bus, you surrender the GPU's massive bandwidth advantage. Everything about how modern deep learning inference engines and data pipelines are architected follows directly from this brutal ratio.
The mechanism › Crossing anyway
Pageable, pinned, and the overlap trick
When you inevitably must cross the bus, the mechanism matters. By default, the CPU allocates pageable memory. But the GPU's Direct Memory Access (DMA) engines cannot safely pull data from pageable host memory, because the operating system could silently relocate those pages mid-transfer. If you command the GPU to read from a pageable array, the driver must secretly allocate a hidden staging buffer, copy the data from the pageable array into the staging buffer using the CPU, and only then transfer it over PCIe.
To bypass this, you allocate pinned (page-locked) memory. Pinning the memory explicitly tells the driver to allocate a buffer that is safe for the DMA engine to read from directly, completely skipping the expensive staging copy.
Even with pinned memory, the PCIe bus remains agonizingly slow. The only way to survive the transfer is to hide it. Because the GPU can execute compute kernels and transfer data simultaneously on separate hardware engines, developers use CUDA streams to overlap the two. A well-structured pipeline chunks the data, streaming chunk B over the PCIe bus at the exact same time the compute units are chewing through chunk A.
ChipDiagram · Level 5 · Interactive
The drinking straw — the system
three pipes, drawn to scale · 52 : 14 : 1 [WP·?]
H100 · 132 SMs
idle
fill 80 GB: HBM ~24 ms · NVLink ~89 ms · PCIe 1.25 s — all derived from [WP] bandwidths
Feed the SMs from
Host memory
Streams
wall-clock
—
timeline · 4 chunks · compute 2 ms/chunk · transfer 1 ms/chunk
amber = bus busy · green = SMs busy · blank = silicon waiting
bandwidths [WP]: HBM3 3,352 · NVLink 900 (total bidirectional) · PCIe 64GB/s per direction — see "what I simplified" · timeline model illustrative (the chapter's 2 ms kernel / 40 ms transfer example; pageable staging +20 ms/chunk) · utilization and wall-clock computed from the lanes, not asserted · NVLink fabric deliberately unexplained
The consequence › Overlap's ceiling
You don't have a fast kernel; you have a slow pipe
Predict first
Your kernel computes a chunk in 2 ms; the chunk takes 40 ms over PCIe. You enable perfect transfer/compute overlap. Speedup?
But we must be honest about what overlap can actually achieve. Transfer times frequently dominate. If you write a beautifully optimized kernel that processes a chunk of data in 2 milliseconds, but that chunk takes 40 milliseconds to arrive over the PCIe bus, overlapping saves you almost nothing. Your 2-millisecond kernel is just a rounding error attached to a bus transfer. You do not have a fast kernel; you have a slow pipe.
The full story › the 0.8 callback — pinned memory means locked pages
When you allocate page-locked memory on the host to speed up GPU transfers, you are directly interacting with the virtual memory concepts from Topic 0.8. By pinning the memory, you are overriding the operating system's right to evict or page out that physical memory to disk. You are telling the OS kernel to make a strict physical exception for your application so the GPU's hardware DMA engine has a physical address that will not move to pull from.
The full story › what I simplified: bidirectionality
In the prose and arithmetic above, I used 64 GB/s for PCIe Gen 5 and 900 GB/s for NVLink. These numbers represent the maximum unidirectional bandwidth for PCIe (64 GB/s out of the total 128 GB/s bidirectional peak) and the total bidirectional bandwidth for NVLink, respectively. This simplification is honest here because data ingestion to feed a compute kernel is fundamentally a unidirectional problem. Giving the PCIe bus credit for its return-trip bandwidth would falsely imply that the drinking straw is twice as wide as it actually is for the task of feeding the fire hose.
We now know the cost of every path in the machine. Can we predict a kernel's runtime before writing it?
The strategy is not "transfer faster" — it is don't transfer. Keep data resident: the straw refills in seconds what the fire hose drains in milliseconds, and no amount of pinning or overlap changes which pipe you're drinking through.