Jul 2026· International Conference on Supercomputing· pp. 662-674· 0 citations· 44 references
Computer Science
TL;DR
ASTRA is a framework that optimizes the holistic streaming CPD process on GPU that integrates the memory pool and asynchronous strategy with compact storage format to reduce idle cycles and execution latency and devises a similarity-aware updating mechanism to reduce the overall workload.
Abstract
Tensors represent multidimensional data and cover various areas of scientific computing. The Canonical Polyadic Decomposition (CPD) emerges to extract latent patterns from large but highly sparse tensors. In real-world scenarios, tensor slices often arrive dynamically over time in streaming form, making traditional CPD algorithms inefficient in processing the entire tensor at each time step. Streaming CPD processes tensor slices incrementally, exploiting a forgetting factor to adjust the weight of historical information to capture dynamics. Current optimizations mainly focus on CPU platforms, failing to meet the real-time processing requirements of modern applications. Efficiently deploying streaming CPD on GPU remains challenging due to frequent data transfers and memory operations throughout the complex workflow, as well as the intricate computational patterns of bottleneck operators. To address these challenges, we propose ASTRA, a framework that optimizes the holistic streaming CPD process on GPU. At the system level, ASTRA integrates the memory pool and asynchronous strategy with compact storage format to reduce idle cycles and execution latency. At the operator level, ASTRA adopts dedicated techniques to customize the kernel implementations based on operator characteristics as well as an analytical model to adjust the parameters. Finally, ASTRA devises a similarity-aware updating mechanism to reduce the overall workload by skipping the calculations of highly similar adjacent time slices. The experimental results show that ASTRA achieves an average speedup of 1.71 × over the state-of-the-art methods with sufficient accuracy.
Modern GPUs increasingly integrate Tensor Cores into the execution pipeline. Although aggregate tensor throughput continues to grow, aided by an operand supply that has evolved from register-based in Ampere to redundancy-free, memory-based in Hopper and Blackwell, efficiently orchestrating the complete tensor compute pipeline for the modern AI workloads remains challenging. We identify the fundamental bottlenecks as fixed parallelism and coarse-grained scheduling, both of which are exposed by modern AI workloads that interleave diverse non-GEMM operations with GEMM. To orchestrate tensor computation efficiently, we propose FIBER, a new architecture that extends the GPU SIMT (single instruction, multiple thread) model. Its basic execution instance, the \emph{fiber}, is decoupled from private register ownership, carrying only minimal control state while accessing an SM's registers through a shared view. This enables dynamic parallelism scaling, fine-grained register-level dataflow scheduling, and offers a redundancy-free alternative for matrix operand supply. We extend the ISA, microarchitecture, and compiler to realize shared-register addressing, conflict-free operand delivery, and fiber-based program mapping. Under a typical mixed-precision LLM serving scenario, FIBER achieves a 2.25x end-to-end speedup on Ampere (1.15x for the original FP16 computation), with 1.8x and 2.09x on Hopper and Blackwell respectively, and kernel-level gains up to 2.49x.
Zihan Liu, Jingwen Leng, Yangjie Zhou et al.· 0 citations
Results show that template-anchored decomposition and stage-specific GPU execution can scale matrix completion beyond device-memory capacity, and show that template-anchored decomposition and stage-specific GPU execution can scale matrix completion beyond device-memory capacity.
Chengying Huan, Yu-Bo Wang, Pin-Huan Wang et al.· 0 citations
ATSInfer combines static tensor placement with load-aware dynamic transfer and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends and can substantially improve the user experience of local LLM deployment on personal consumer devices.
Yangyijian Liu, Hong-Yi Ye, Ming-Yang Li et al.· arXiv.org· 0 citations
A unified ACS2 framework for comparing five execution regimes: a symbolic baseline, single-process CPU execution, multiprocessing CPU execution, sequential independently seeded tensorized GPU execution, and the same PyTorch tensor backend executed on the host CPU.
It is found that no single kernel dominates across all graph structures and that effective Tensor Core utilization reaches only 5–20% on irregular GNN matrices, while graph reordering is broadly beneficial, yielding gains of up to $43\times $ when it enables Sparse Tensor Core execution.
Tariq Habib Afridi, Muhammad Numan Khan, Young-Koo Lee· IEEE Access· 0 citations
Sparse GEneral Matrix Multiplication (SpGEMM) is one of the most vital kernels in massive research domains, including bioinformatics, graph analytics, and machine learning. Moreover, with the prosperity of the Big Data era, nonzero elements in sparse matrices of SpGEMM boost rapidly into the magnitude of billions. Thus, accelerating billion-scale SpGEMMs becomes the new spotlight. To reduce both the monetary and energy costs of resource-intensive billion-scale SpGEMMs while enhancing their efficiency, we propose CESpGEMM, an adaptive resource-efficient library for billion-scale SpGEMMs on CPU-GPU heterogeneous servers. We design an Anchor-based Sparsity-aware Data Compression (ASDC) mechanism of high compression ratio and throughput to resolve the bottleneck of data transfer through PCIe. Moreover, given an arbitrary SpGEMM, we develop a sampling-based auto-tuning model to determine the optimal hyperparameter configuration. CESpGEMM also involves a hierarchical workflow that exploits adaptive CPU-GPU collaboration. We evaluate the performance of CESpGEMM using one T4 GPU with state-of-the-art SpGEMM libraries. The results demonstrate that compared to the cutting-edge implementation with 4 T4 GPUs, CESpGEMM outperforms these in terms of memory and communication efficiency, and can achieve the average acceleration rate of 5.42×.
Ming Dun, Cheng Zhang, Shuhan Song et al.· IEEE Transactions on Paralle...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
