Oct 2026· IEEE Transactions on Parallel and Distributed Systems· Vol 37, pp. 2205-2218· 0 citations· 48 references
Computer Science
Abstract
Deep neural networks (DNNs) with billions of parameters power many important applications, but their training is fundamentally constrained by the limited on-chip memory of GPUs. This memory wall forces training to rely on distributed execution or memory offloading, both of which introduce substantial inefficiencies. Existing offloading techniques can scale model size but often incur severe throughput degradation, while conventional distributed training suffers from poor hardware utilization due to limited cross-device bandwidth, leaving accelerator resources underused. We present SynergyScale, a software framework that addresses the GPU memory wall by jointly optimizing memory usage and computation efficiency. SynergyScale combines dynamic hierarchical memory offloading across GPUs, CPU, and secondary storage tiers to reduce GPU memory pressure with fine-grained multi-stream task partitioning that takes advantage of the concurrent execution capabilities of modern GPUs. The framework automatically selects offloading parameters that balance data movement overhead with parallel execution efficiency. Experimental results show that SynergyScale can train a 114.41B-parameter model on a single 80 GB NVIDIA A100 GPU and a 547.77B-parameter model on eight A100 GPUs, supporting models up to 29.5× larger than those enabled by existing approaches. By co-optimizing offloading and computation, SynergyScale achieves up to 4.5× higher training throughput than offloading-only methods, without altering training semantics, making billion-scale model training both efficient and cost-effective.
To train or fine-tune large language models with insufficient GPU memory, heterogeneous parallel training methods utilize aggregated GPU memory and offload tensors to CPU DRAM or SSD. However, these methods lack effective simultaneous management of GPU and CPU memory, creating a critical bottleneck on resource-constrained commodity servers since insufficient capacity in either component leads to training failure. Moreover, existing methods incur excessive GPU memory usage without corresponding performance gains. To address these limitations, we propose a heterogeneous pipeline parallelism scheme named Tiny-Pipe that efficiently utilizes both GPU and CPU memory while maintaining comparable or superior performance. Tiny-Pipe comprises three key components: (1) a holistic layer packing method that simultaneously reduces GPU memory footprint and improves training performance, (2) active CPU memory management that alleviates CPU memory pressure by eliminating redundant parameters, and (3) a layer-wise runtime swapping strategy that further enhances overall performance. Experimental results demonstrate that our approach achieves (1) the smallest GPU and CPU memory footprint across all cases and (2) the broadest training coverage—successfully training all model configurations—while (3) maintaining optimal performance in most scenarios.
General Matrix Multiplication (GEMM) is the cornerstone of high-performance computing and deep learning. Its efficiency significantly influences the performance of applications ranging from large language models to scientific simulations. Intel Advanced Matrix Extensions (AMX) significantly boost matrix operations throughput, yet existing implementations underutilize its potential. The core bottleneck is that AMX’s unique microarchitectural constraints break the classic Advanced Vector Extensions 512 (AVX-512) GEMM paradigm, causing critical data reuse loss across the memory hierarchy: inefficient vector register reuse during mandatory layout transformation, poorly reconciled trade-off between Tile register reuse for partial sums and L1 cache locality, and severe buffer reuse degradation due to fine-grained update strategies. To address these challenges, we propose TileGEMM, a high-performance GEMM implementation on AMX that systematically enhances data reuse across the memory hierarchy. First, a dual Tile register layout transformation strategy eliminates redundant memory accesses, maximizing vector register reuse. Second, a multi-level cache- and Tile register-aware tiling strategy adopts a K-dimension-centric model under strict L1 capacity constraints, improving Tile register reuse for intermediate partial sums while preserving L1 cache locality. Third, a coarse-grained buffer update strategy with loop reordering extends transformed data residency, boosting buffer reuse and reducing transformation overhead. Extensive experiments in various matrix configurations show that TileGEMM achieves average speedups of 3.27 × and 1.27 × over AMX-based implementations LIBXSMM and OneDNN, and 10.30 × and 7.96 × over AVX-512-based implementations TVM and MKL, respectively.
Kang-Kang Chen, Hua-You Su, Meng-Han Jia et al.· 0 citations
This paper proposes LayUp, a layer-wise training optimization framework for edge devices with unified memory architectures that achieves speedup and energy reduction compared to baseline GPU-only training for GPT-2 models on the NVIDIA Jetson Orin NX, while conventional offloading increases latency.
Bang-San Lee, Young-Ho Gong· International Symposium on L...· 0 citations
LazyTrain is proposed, an optimization layer over a layer-streaming executor that formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training.
Xiao-Jun Wu, Cehao Yang, Honghao Liu et al.· 0 citations
This work proposes SAI, a mechanism that virtualizes shared memory into the L2 cache to improve GPU performance for AI applications and introduces an L2 cache management strategy that integrates associativity-based virtual page allocation and a replacement information table, reducing page-swapping overhead while preserving L2 cache performance.
Hanqing Li, Tiejun Li, Sheng Ma et al.· ACM Transactions on Design A...· 0 citations
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
Jing-Hao Wang, Yi-Hang Zhou, Xiaoyang Sun et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
