Aug 2026· Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication· pp. 505-518· 0 citations· 34 references
Computer Science
TL;DR
This work presents Meta's network architecture and software stack designed to support one of the world's largest RoCE fabrics, currently connecting over 100,000 GPUs across multiple datacenter buildings, and introduces a scalable initialization strategy that reduces startup times by 11× via eager process group creation and O(N) topology discovery.
Abstract
The arrival of 100K+ GPU clusters marks a new frontier in AI infrastructure. Standard communication stack meets new challenges as physical topologies span multiple datacenter buildings, introducing high bandwidth-delay product links where latency increases by up to 30× compared to intra-rack traffic. Furthermore, the transition toward Mixture-of-Experts architectures generating bursty all-to-all patterns that create transient congestion hotspots. These constraints, combined with an operational environment where hardware failures shift from anomalies to frequent occurrences, renders traditionally lightweight operations like initialization and resource management challenging. We present Meta's network architecture and software stack designed to support one of the world's largest RoCE fabrics, currently connecting over 100,000 GPUs across multiple datacenter buildings. To overcome scaling barriers, we introduce a scalable initialization strategy that reduces startup times by 11× via eager process group creation and O(N) topology discovery, alongside a resource management system that cuts GPU memory usage by 2× through on-demand allocation. We further detail a custom transport layer utilizing Dynamic Queue Pair Load Balancing to saturate links, and a set of operation toolings. These innovations have been deployed in production, providing the foundational communication fabric for training state-of-the-art Large Language Models.
Building on NCCL's device-side API, low-latency interfaces for constructing custom collective kernels are developed and used to implement new symmetric collectives in NCCL, demonstrating benefits for both AI inference and traditional HPC workloads.
Siyuan Shen, Anton Korzh, J. Bachan et al.· arXiv.org· 0 citations
This work highlights the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies and introduces NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic.
Large Language Model (LLM) training is increasingly concentrated in homogeneous datacenters, while private data and underutilized GPUs across universities, laboratories, and edge sites remain difficult to use. This extended abstract presents preliminary results from a geo-distributed LLM training prototype that treats networking constraints as first-order design concerns. The prototype connects three heterogeneous GPU sites via cloud-hosted parameter servers, outbound-only gRPC streams, two-stage delta compression (INT8 quantization + Huffman coding, achieving up to 4× payload reduction), and fault-tolerant rejoin. In real deployments, GPT-2 Medium pretraining achieves stable loss reduction and reaches the target loss 15.2% faster in wall-clock time than the best tested baseline; Llama3-1B pretraining remains stable under larger communication pressure; and cross-site latency traces reveal site-dependent WAN spikes of up to 200s. These results motivate adaptive networking support for synchronization, compression, placement, telemetry, and recovery in geo-distributed LLM training.
Ziyue Luo, Jiaxuan Cai, Cedric Le Denmat et al.· Conference on Applications,...· 0 citations
This work presents CCSwitch, a modular switching fabric built from 4×4 non-blocking Collective Engines, a modular switching fabric built from 4×4 non-blocking Collective Engines (CEs) that combines spatial and temporal parallelism to perform reductions without accumulation buffers.
Sumukh Pinge, Hardik Soni, Bob Lantz et al.· Conference on Applications,...· 0 citations
The rapid growth of distributed machine learning systems shifts ML networks from the CPU/GPU-NIC to GPU-Fabrics architectures, creating an emerging need to measure and optimize NIC-less Scale-up networks. Yet, the disappearance of NICs removes key observability points and leaves existing measurement tools largely limited to coarse, application-layer telemetry. We present FabricPerf, a Scale-up network measurement toolkit that enables fine-grained packet-level timestamping and cross-layer hardware insights. FabricPerf uses a software-hardware co-design that integrates protocol-level probing and GPTP clock synchronization within GPU communication kernels with memory profiling of physical pipelines. We implement FabricPerf for the CUDA/NvLink ecosystem and validate its high precision and low overhead via extensive experiments on H100 and GB200 NVL72. Using FabricPerf, we explore Scale-up network dynamics under varied configurations, with two case studies that reveal critical bottlenecks: (i) channel imbalance in high P99 latency and (ii) memory pipeline inefficiency of ~0% LLC hit-rate. To mitigate these issues, we leverage work-stealing to rebalance channels by 46% and boost throughput by 17.5GB/s, and optimize LLC eviction priorities for memory flows to elevate the LLC read hit-rate to ~35%, yielding 16.07GB/s bandwidth improvement. We believe FabricPerf improves Scale-up network observability, laying a foundation for more complete and efficient network stacks. FabricPerf is open-sourced at github.com/open-neutrino/fabricperf.
Song-Ling Huang, Chenshu Wu· Conference on Applications,...· 0 citations
StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features.
Tian-Cheng Hu, Jin Qin, Yu-Zheng Wang et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
