2026· IEEE Transactions on Networking· Vol 34, pp. 8011-8026· 0 citations· 48 references
TL;DR
It is proved that for non-oversubscribed networks, there exists an optimal communication scheduling scheme capable of completely avoiding bandwidth contention among the scheduled flows in each collective-communication stage and achieves up to 77.5% reduction in iteration time compared to state-of-the-art methods.
Abstract
The rapidly growing demand for distributed training has motivated providers to deploy large-scale GPU clusters. As a provider, we manage multiple clusters that hosts numerous distributed model training jobs that generate voluminous, highly dynamic concurrent flows. These flows may traverse the same link, causing severe bandwidth contention. Existing efforts attempt to minimize bandwidth contention by selecting least-congested links for each flow and prioritizing high-priority flows, yet they fail to eliminate it entirely. Our analysis shows that even a minimal number of flows under bandwidth contention may still cause severe straggler problems, significantly impairing communication time. To bridge this gap, we design OpDisPath, an approach that selects link-disjoint paths for the collective flows in each communication stage. Specifically, most GPU clusters are built on non-oversubscribed networks, i.e., the total uplink bandwidth of a switch is no less than the total downlink bandwidth. We prove that for such networks, there exists an optimal communication scheduling scheme capable of completely avoiding bandwidth contention among the scheduled flows in each collective-communication stage. To achieve this goal, we devise a disjoint path selection algorithm based on bipartite graph matching and provide theoretical proofs of its algorithmic optimality. Experimental validation on a physical testbed and through large-scale simulations of a 2,016-GPU cluster demonstrates that OpDisPath eliminates all bandwidth contention within each communication stage and achieves up to 77.5% reduction in iteration time compared to state-of-the-art methods.
VarioPath, an efficient AlltoAllv scheduling framework for PCIe GPU systems that combines an offline topology-aware analyzer with an online demand-aware scheduler, and shows average AlltoAllv speedups of 5.88x over FAST and 1.72x over DeepEP.
Yao Fei, Jin Fang, Si-Ze Zheng et al.· arXiv.org· 0 citations
AI training workloads generate structured, predictable traffic that alternates between computation and synchronized collective operations, yet existing network schedulers do not fully exploit this structure for scheduling. Reconfigurable networks can exploit this structure by dynamically adapting connectivity to commun...
Teja Nikhil Masabattula, N. Soni, Sidharth Sharma· Proceedings of the ACM CoNEX...· 0 citations
OpenRDMA is proposed, an open-source multi-path reliable RDMA transport architecture designed for GPU-friendly communication and describes the architecture and key hardware components of OpenRDMA.
Zi-Heng Ji, Chun-Yu Liu, Wei-Hao Peng et al.· 0 citations
DualColl is presented, which targets bandwidth-bound AllReduce directly and extends the fractional-packing framework to additional collective operations and multiple concurrent collective groups, and introduces a tree-coupled max-min bandwidth-sharing model.
Cadence targets multi-tenant LLM serving over operator-declared phases on disaggregated GPU pools and remote KV/cache tiers. We assume a small phase taxonomy—retrieval, tool fan-out, structured parsing, and decode—and adapt resource parameters within each phase online, rather than discovering arbitrary new behaviors at...
Rui Li, Shuang Cao· Proceedings of the 2026 ACM...· 0 citations
The core of LEO lies in introducing hierarchical caching to construct a hybrid-grained software pipeline, enabling efficient computation-communication overlap both within and across GPU kernels.
Jia-Qi Si, De-Zun Dong· ACM Transactions on Architec...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.