A transformer architecture where the MLP layer is split into many "experts" and a router activates only a small fraction per token, decoupling total parameters from compute per token.
Tensions
Going sparser lowers compute but inflates total parameters super-linearly for quality gains (e.g. 64x params for 4x effective dense size), and pushes more onto memory bandwidth and scale-up bandwidth. The sweet spot is "as sparse as your scale-up domain can hold the weights and your traffic can fill the batch."
Related Concepts
parallelism | scale-up domain | batching | memory bandwidth | inference economics