The buildings and cabling that large models are trained in. A frontier training run is not one machine but tens of thousands of chips wired together, and how fast they can talk to each other frequently matters more than how fast each one calculates. This is why these facilities are so expensive, so concentrated, and increasingly constrained by the availability of electricity.
Technical view
Training is distributed across nodes using data, tensor and pipeline parallelism, all of which require frequent synchronisation of gradients or activations, making interconnect bandwidth and latency a first-order constraint. Technologies such as NVLink within a node and InfiniBand or high-speed Ethernet between nodes define achievable scale, and communication cost is why realised efficiency against theoretical peak is often well under half. At these scales hardware failures are routine rather than exceptional, so checkpointing and automatic restart are core parts of the training stack.
Related
Part of AI Terminology Index, section: Hardware, Compute and Training Mechanics.