The set of ways to spread a model and its activations across many chips: data, tensor, expert, and pipeline parallelism — each cutting along a different dimension of the model.
Tensions
The empirical winners (expert within a rack, pipeline across racks) match the model's natural seams, while the historically beloved tensor parallelism has faded as experts have shrunk. The choice is constrained by which communication pattern fits which network: all-to-all wants full intra-rack connectivity, pipelining tolerates slow scale-out.
Related Concepts
mixture of experts | pipelining | scale-up domain | memory bandwidth