Skip to content
Open access

Canvas+: Scalable Collective Communication Scheduling for Large-Scale GPU Clusters

2026 · IEEE Transactions on Networking · Vol 34, pp. 7965-7980 · 0 citations · 39 references

Abstract

State-of-the-art deep learning models rely on large GPU clusters and various parallelism strategies, which in turn depend on collective communication (CC) operators to synchronize data. While vendor libraries (e.g., NCCL, RCCL) provide standard CC algorithms, they often suffer from bandwidth bottlenecks in modern, complex topologies. Recent synthesis-based methods improve performance but face four fundamental limitations: the combinatorial explosion of the scheduling space, monolithic execution prone to transient congestion, bandwidth waste from sequential micro-batches, and stranded inter-group bandwidth under strict closed-group routing in spatially multiplexed clusters. We propose Canvas+, a scalable and near-optimal CC scheduling framework that overcomes these architectural bottlenecks. Canvas+ introduces four key innovations: 1) Hierarchical synthesis with semantic subgraph stitching to decompose the global scheduling problem and accelerate synthesis by reusing verified schedules. 2) Collective decomposition with symmetry-induced load balancing to enable structured, multi-stage, and congestion-free algorithm generation. 3) Cross-micro-batch pipeline scheduling to eliminate sequential execution dependencies and maximize concurrent link utilization. 4) Fragmented resource integration via relay routing to reclaim stranded scale-up bandwidth by routing flows through idle, non-participating nodes. Extensive evaluations show that Canvas+ achieves up to $1.98\times $ and $3.56\times $ bandwidth speedups over TACCL and TE-CCL, respectively, while reducing synthesis time by 3–4 orders of magnitude at large scales. Relay-assisted routing further compresses scheduling epochs by up to 44.4% in spatially multiplexed clusters.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.