Book
Open access
Aug 2026
Connecting 100K+ GPUs: Building the Communication Stack for Large-Scale LLM Training
This work presents Meta's network architecture and software stack designed to support one of the world's largest RoCE fabrics, currently connecting over 100,000 GPUs across multiple datacenter buildings, and introduces a scalable initialization strategy that reduces startup times by 11× via eager process group creation and O(N) topology discovery.
Hongyi Zeng, Min Si, Pavan Balaji et al.
· Conference on Applications,... · 0 citations