A back-of-envelope model that estimates the runtime of a workload as the max of compute time and memory-fetch time, ignoring everything else, and turns out to be surprisingly predictive for transformer inference.
Tensions
The model is intentionally crude: a real cluster has all-to-all communication overhead, MFU losses, and prefill/decode interleaving that the simple max ignores. The right reaction is to add 2-3x safety margin, not to add detail — most insight lives in where the two curves cross.
Related Concepts
inference economics | batching | memory bandwidth | mixture of experts