The rate at which a GPU can move bytes between HBM and the compute units — typically the binding constraint on LLM inference latency and on practical context length, far more often than raw FLOPs.
Tensions
Bandwidth scales with the number of chips in a scale-up domain, not with chip count alone, so adding racks via scale-out helps capacity but not latency. Pipelining hides weight bandwidth across racks but cannot do the same for KV cache bandwidth, which is why long context remains expensive.
Related Concepts
scale-up domain | KV cache | latency | batching | roofline analysis | inference economics