The maximum number of tokens a model can attend to in one forward pass — practically capped not by training tricks but by KV-cache memory bandwidth at serving time.
Tensions
Frontier context has been stuck near 100-200K for two years because that is where the cost-equal point sits for dense attention. The "we just need bigger context for AGI" claim runs straight into this bandwidth wall; without sparse attention, getting to hundred-million-token contexts is not affordable.
Related Concepts
KV cache | sparse attention | memory bandwidth | inference economics