The set of GPUs/TPUs connected by the fast, fully-connected scale-up network (NVLink/NVSwitch on Nvidia, ICI on TPUs), inside which all-to-all communication is cheap — typically one rack on Blackwell (72 GPUs), much larger historically on TPU pods.
Tensions
Bigger is better for both bandwidth and the size of expert-parallel layer you can fit, but physical constraints (cable density, power, weight, cooling) cap how many chips fit in one rack. The real engineering battle isn't FLOPs per chip, it's wire density per rack.
Related Concepts
memory bandwidth | parallelism | mixture of experts | hardware