The cached key/value tensors of previously processed tokens that an autoregressive decoder attends to — the part of inference memory that scales with batch size and context length, and which neither batching nor pipelining can amortize away.
Tensions
You can store it in HBM (fast, expensive), in DDR/flash (slow, cheap), or recompute it from scratch (cache miss cost ≈ a full prefill pass). The choice depends on how long you intend to keep it. Long context is bottlenecked here, not in compute, which is also why million-token windows have not arrived without sparse attention.
Related Concepts
memory bandwidth | batching | pipelining | sparse attention | context length | inference economics