StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features.
Abstract
Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries remain largely buffer-centric because user and communication buffers are managed separately, causing redundant data copies or costly user-buffer registration. This paper presents StrataCL, a zero-redundancy and fabric-native communication library for production supernodes. StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features. On the Huawei CloudMatrix384, StrataCL improves collective bus bandwidth by up to 1.6x and improves MoE dispatch/combine bus bandwidth by up to 1.4x. Across three production workloads, StrataCL improves LLM inference throughput by 1.9x, reduces P99 TTFT by 2.2x, and reduces LLM and Recsys training iteration time by 1.4x and 1.3x, respectively.
This work presents CCSwitch, a modular switching fabric built from 4×4 non-blocking Collective Engines, a modular switching fabric built from 4×4 non-blocking Collective Engines (CEs) that combines spatial and temporal parallelism to perform reductions without accumulation buffers.
Sumukh Pinge, Hardik Soni, Bob Lantz et al.· Conference on Applications,...· 0 citations
LLM inference has become an essential service, yet it imposes unprecedented demands on memory bandwidth, computational density, and communication efficiency. While IMC is a promising solution to the memory wall issue, the heterogeneous data dynamicity of LLM requires complementary resources to handle intermediate data generated during run-time. Furthermore, the massive number of parameters in LLM necessitates scale-up architectures where on-chip data movement is often the primary performance bottleneck. This article presents a hardware-software co-design framework that unifies distributed compute, memory, and communication into a seamless processing-communication fabric. On the hardware side, we propose a scalable architecture, named LEAP, that integrates IMC PE, NMC PE, and INC. This allows each hardware layer to execute specialized tasks: IMC for static weights, NMC for dynamic data, and INC for partial result reduction. On the software side, we introduce a partitioning, mapping, and scheduling framework optimized for key metrics in LLM serving, including throughput and latency. To address the distinct computational intensities of the prefill and decode phases, we present a prefill-decode disaggregation approach that dynamically reconfigures PE organizations to maximize resource utilization. Compared to commercial GPU platforms, the proposed architecture provides a throughput and an energy efficiency improvement of $\geq{}1.52\times$ and $24.91\times$, respectively.
Modern collective communication libraries (CCLs) execute a collective communication task (CCT) by decomposing it into multiple sub-tasks, each mapped to a specific Virtual Topology (VT), which is an ordered graph of GPUs (e.g., a ring or a tree), to maximize parallelism and link utilization. As AI training scales to larger clusters, network anomalies (congestion and failures) are unavoidable, and a single straggling VT can delay the entire CCT. Existing solutions either rely on low-level transport-layer solutions which lacks a cross-sub-task perspective, or static CCL scheduling, failing to adapt to the dynamic and heterogeneous networks. We present PReCCL, a drop-in NCCL replacement that combines software inband telemetry with cross-VT workload reallocation. Unlike hardware-dependent telemetry (e.g., P4-INT), PReCCL implements in-band monitoring within the CCL. It precisely measures the stall counts of each VT, and piggybacks the telemetry meta-data on existing collective traffic. With this fine-grained visibility, PReCCL employs a dynamic cross-VT reallocation mechanism that shifts workloads from straggler VTs (suffering from congestion or faults) to healthy ones. Reallocation occurs only at CCT boundaries using an epoch-based deterministic protocol, avoiding cross-rank consistency races. We implement PReCCL in NCCL and evaluate it on a 32-GPU testbed, large-scale simulation, and a 1024-GPU production cluster. On the testbed, PReCCL improves CCT completion by up to 2.1X and end-to-end training by 1.21X, with low overhead above tuned message-size thresholds.
Zhiyong Chen, Kaihui Gao, Li Chen et al.· Conference on Applications,...· 0 citations
STORM is presented, a NIC-level scheduler for all types of RDMA workloads using NIC-only information: the known RDMA request size, and per-queue-pair backlog, and converts these signals into a small number of extra priority levels on the wire and prioritizes requests that are either near completion or blocking queued dependent work.
Jichun Wu, Ran Shu, Gianni Antichi et al.· Conference on Applications,...· 0 citations
Multi-LoRA serving is how one base model becomes thousands of specialized variants, one adapter per user, task, or agent, and the deployments can hold 1000-plus adapters. Serving them is hard because the workload inverts what GPUs provide: terabytes of memory against only tens of TFLOPS, and because every published system stages its adapters from CPU DRAM over PCIe, where each access pays a kernel stop and a host-run copy and capacity ends at the motherboard's DIMM slots. Meanwhile, memory-semantic fabrics such as CXL and NVLink are converging on pooled memory that an accelerator addresses with its own loads and stores, and near-data processing (NDP) can place compute beside the pooled data. How to serve multi-LoRA workloads on such hardware remains unexplored. This paper introduces PLoRA, an NDP-enhanced pooled-memory system for cost-efficient multi-LoRA serving. PLoRA keeps adapters and KV cache in the pool and returns only reduced results over the link, through a read-compute interface the GPU drives with its own loads and stores. Above this architecture, a GPU memory management system picks among four LoRA and two attention execution strategies for each adapter and caches the most performance-critical bytes in GPU memory, guided by a link-parameterized cost model. On one H100 serving 1000 adapters, PLoRA attains the lowest decode latency on every model and workload we measure, averaging 6.6x below a real-machine S-LoRA at under 3.4% added device area. The link itself stops mattering: throughput saturates at 32 GB/s on short contexts, a quarter of CXL 3.1, and the verdict survives scale: per-GPU demand falls from 7B to a modeled 1.2T deployment once adapter traffic shards with the tensor parallelism. The design runs unchanged from CXL-class to NVLink-class fabrics, and surplus bandwidth buys pooled capacity rather than speed.
Zhongkai Yu, O. Venkatachalam, Zheng Wang et al.· 0 citations
This paper presents the first systematic study showing that memory contention, not network bandwidth, is the dominant bottleneck in message aggregation runtimes, and advocates for dedicated near-memory subsystems to improve the scalability and performance of message aggregation runtimes.
Shubhendra Pal Singhal, Aaron Welch, Oscar Hernandez et al.· IEEE International Symposium...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
