Running many independent inference requests through the same forward pass so the cost of fetching weights is amortized across users — the single most important lever in inference economics.
Tensions
Without batching, cost-per-token is roughly 1000x worse. But the batch is also a queue: it forces a fixed cadence (~20ms on modern HBM), so a large batch buys cheaper tokens at the price of tail latency for any individual request.
Related Concepts
inference economics | memory bandwidth | latency | roofline analysis | mixture of experts