GPU memory is not one thing. A useful first split is:
- Registers: closest to execution, private to a thread.
- Shared memory: explicitly managed memory shared inside a thread block.
- L2 cache: shared across SMs and often the last on-chip stop before HBM.
- HBM/global memory: high bandwidth, high latency relative to on-chip storage.
The core optimization question is how many times a byte is reused before it leaves the faster levels. GEMM kernels are mostly elaborate machines for increasing that reuse.
Related course note: WGMMA 与异步执行范式.