This work highlights the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies and introduces NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic.
Abstract
Graphics processing unit (GPU) architectures are growing in size to meet the increasing compute and memory requirements. As GPU sizes increase, intra-socket wire transfer delay increases significantly. While previous research has optimized for compute and memory locality within a socket, the spatial impact on inter-GPU communication has not been well-studied. We introduce the term non-uniform network access (NUNA) to describe this emerging optimization dimension in multi-GPU systems. We specifically focus on latency-sensitive collective communication, common in machine learning inference. First, we highlight the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies. Second, we introduce NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic. We demonstrate that the NAP optimizations alone offer up to 1.5x collective speedups over a locality-unaware baseline. Combining NAP with NAR yields up to 1.8x faster collectives over the locality-unaware baseline. This leads to 7% mean (28% max) time per output token speedup in machine learning inference.
Building on NCCL's device-side API, low-latency interfaces for constructing custom collective kernels are developed and used to implement new symmetric collectives in NCCL, demonstrating benefits for both AI inference and traditional HPC workloads.
Siyuan Shen, Anton Korzh, J. Bachan et al.· arXiv.org· 0 citations
This work categorizes LLM kernel operands into three inter-workgroup sharing patterns and shows that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling, highlighting the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.
Donghyeon Joo, Sooraj Puthoor, N. Jayasena et al.· arXiv.org· 1 citation
Gleam is a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions that reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution.
SA-Scheduler is presented, a structure-derived bottleneck-aware scheduling framework for multitasking MCM-GPUs that determines chip placement without hardware modification or runtime bottleneck profiling, and provides a principled and scalable foundation for multitasking on future MCM-GPUs.
Tiejian Zhang, Guangda Zhang, Lu Wang et al.· ACM Transactions on Design A...· 0 citations
A time-varying integer program to minimize the long-term total cost of the edge AI inference system, including the inference latency, the inference error rate, the query-dispatching communication cost, and the energy consumption, subject to resource and workload constraints is proposed.
Mingtao Ji, Hehan Zhao, Lei Jiao et al.· Science China Information Sc...· 0 citations
The rapid growth of distributed machine learning systems shifts ML networks from the CPU/GPU-NIC to GPU-Fabrics architectures, creating an emerging need to measure and optimize NIC-less Scale-up networks. Yet, the disappearance of NICs removes key observability points and leaves existing measurement tools largely limited to coarse, application-layer telemetry. We present FabricPerf, a Scale-up network measurement toolkit that enables fine-grained packet-level timestamping and cross-layer hardware insights. FabricPerf uses a software-hardware co-design that integrates protocol-level probing and GPTP clock synchronization within GPU communication kernels with memory profiling of physical pipelines. We implement FabricPerf for the CUDA/NvLink ecosystem and validate its high precision and low overhead via extensive experiments on H100 and GB200 NVL72. Using FabricPerf, we explore Scale-up network dynamics under varied configurations, with two case studies that reveal critical bottlenecks: (i) channel imbalance in high P99 latency and (ii) memory pipeline inefficiency of ~0% LLC hit-rate. To mitigate these issues, we leverage work-stealing to rebalance channels by 46% and boost throughput by 17.5GB/s, and optimize LLC eviction priorities for memory flows to elevate the LLC read hit-rate to ~35%, yielding 16.07GB/s bandwidth improvement. We believe FabricPerf improves Scale-up network observability, laying a foundation for more complete and efficient network stacks. FabricPerf is open-sourced at github.com/open-neutrino/fabricperf.
Song-Ling Huang, Chenshu Wu· Conference on Applications,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
