Oct 2026· IEEE Transactions on Parallel and Distributed Systems· Vol 37, pp. 2193-2204· 0 citations· 37 references
Computer Science
Abstract
Sparse GEneral Matrix Multiplication (SpGEMM) is one of the most vital kernels in massive research domains, including bioinformatics, graph analytics, and machine learning. Moreover, with the prosperity of the Big Data era, nonzero elements in sparse matrices of SpGEMM boost rapidly into the magnitude of billions. Thus, accelerating billion-scale SpGEMMs becomes the new spotlight. To reduce both the monetary and energy costs of resource-intensive billion-scale SpGEMMs while enhancing their efficiency, we propose CESpGEMM, an adaptive resource-efficient library for billion-scale SpGEMMs on CPU-GPU heterogeneous servers. We design an Anchor-based Sparsity-aware Data Compression (ASDC) mechanism of high compression ratio and throughput to resolve the bottleneck of data transfer through PCIe. Moreover, given an arbitrary SpGEMM, we develop a sampling-based auto-tuning model to determine the optimal hyperparameter configuration. CESpGEMM also involves a hierarchical workflow that exploits adaptive CPU-GPU collaboration. We evaluate the performance of CESpGEMM using one T4 GPU with state-of-the-art SpGEMM libraries. The results demonstrate that compared to the cutting-edge implementation with 4 T4 GPUs, CESpGEMM outperforms these in terms of memory and communication efficiency, and can achieve the average acceleration rate of 5.42×.
Sparse matrix-dense matrix multiplication (SpMM) is a core operation in scientific computing and deep learning. On multi-GPU platforms, its scalability is limited by communication bottlenecks. To address this, we propose DistSpMM, a co-design framework integrating data layout, pipelining, and communication strategies. DistSpMM introduces HSDMA, a lightweight algorithm that reduces communication by optimizing the dense matrix allocation. DistSpMM features a topology-aware two-stage pipeline that manages the IB/NVLink bandwidth disparity to maximize the overlap of computation and communication. Finally, DistSpMM employs an adaptive selector that uses a performance model to dynamically choose the optimal communication granularity (coarse vs. fine-grained) based on data sparsity and network tier. Experiments on diverse real-world datasets demonstrate the superior performance of our method. It achieves average speedups of 1.6 × to 2.6 × in single-node multi-GPU environments and 4.0 × to 5.1 × in multi-node multi-GPU environments over the baseline. Compared to the best state-of-the-art implementations, our method delivers up to 2.0 × speedup.
Junyu Gu, Jue Wang, Zhikuang Xin et al.· ACM Transactions on Architec...· 0 citations
It is found that no single kernel dominates across all graph structures and that effective Tensor Core utilization reaches only 5–20% on irregular GNN matrices, while graph reordering is broadly beneficial, yielding gains of up to $43\times $ when it enables Sparse Tensor Core execution.
Tariq Habib Afridi, Muhammad Numan Khan, Young-Koo Lee· IEEE Access· 0 citations
Sparse Matrix-Matrix Multiplication (SpMM) is a fundamental operation in scientific computing and artificial intelligence. However, the inherent sparsity and irregularity of real-world datasets present significant challenges for developing high-performance SpMM kernels on modern GPUs. Existing approaches typically focus on a single type of compute unit, leaving the potential for cooperative parallelism between heterogeneous GPU cores largely under-explored. This paper introduces CoTC-SpMM, a cooperative Tensor–CUDA cores scheme for efficient SpMM on GPUs. Specifically, we first propose the HTC format to partition sparse matrices into dense and sparse components, enabling specialized kernels to leverage the distinct advantages of different computing units and maximize hardware utilization. Moreover, we implement several low-level runtime optimizations, including 1-D resource mapping for load balancing, software pipelining to hide memory latency, and PTX-level instruction tuning to enhance SpMM throughput. Experimental results on NVIDIA A100 and H800 GPUs demonstrate that CoTC-SpMM achieves substantial performance speedups over state-of-the-art implementations.
Qi Du, Shengle Lin, Yuedan Chen et al.· Proceedings of the Internati...· 0 citations
AlphaFold 3 (AF3) enables atomic-resolution prediction of biomolecular complexes, driving rapidly growing demand across the life sciences. However, its ∼ 750,GB reference database has effectively confined production deployments to systems with shared parallel filesystems, creating a major barrier for scalability. Distributed high-throughput computing (dHTC) platforms offer vast, heterogeneous compute capacity, but fundamentally lack the shared data infrastructure assumed by AF3. We present a data-aware deployment of AF3 for dHTC, implemented on the Center for High Throughput Computing (CHTC) and the Open Science Pool (OSPool). The workflow is decomposed into a CPU-bound data pipeline that executes on nodes with locally staged, scheduler-advertised databases, and a GPU-bound inference pipeline that opportunistically scales across distributed resources. Using CUDA Unified Virtual Memory (UVM), we extend inference beyond physical GPU limits, enabling predictions of ultra-large complexes that exceed device vRAM. By elevating dataset locality to a schedulable resource via HTCondor ClassAds, we eliminate prohibitive per-job data transfers and enable efficient, federated execution. Beyond scaling throughput, we demonstrate that dHTC can support previously infeasible workloads. Together, these results establish dHTC as a viable—and in some regimes superior— execution model for data-intensive structural biology workflows and provide a general blueprint for deploying large, data-intensive applications on distributed cyberinfrastructure.
Daniel A Morales, Brian Lin, M. Rynge et al.· Practice and Experience in A...· 0 citations
Large linear programs can exceed the memory of a single compute node. Although first-order methods replace sparse factorizations with GPU-suited matrix-vector products, other solver phases can reintroduce a single-node memory limit. We present SHARDLP, a distributed GPU LP solver that keeps the matrix and primal-dual state partitioned from sharded input through solution output. On the Google PDLP benchmark, SHARDLP reaches the published criterion on nine of eleven instances, compared with eight in the published CPU PDLP study. On the largest benchmark, eight H200 GPUs solve a 1.185-billion-variable, 6.338-billion-nonzero LP in 9.9 minutes; the published CPU experiment reports 21.06 hours on different hardware. Beyond this benchmark, separately checked multi-node solves reach up to 13.604 billion variables and 40.807 billion nonzeros, while validated executions span up to 76 GPUs across 29 compute nodes. For column-partitioned solves, support-aware communication skips GPUs that store no coefficients for a row; on an LP with 2.76 billion nonzeros, it cuts modelled communication by 92.97% and improves solver time by 1.27x-1.52x
Arnaud Deza, S. Dey, P. Van Hentenryck· 0 citations
Matrix multiplication is a fundamental computation kernel in many parallel and sequential scientific applications. We target FP32 matrix multiplication on GPUs, a setting required by numerous HPC and scientific workloads. Alternative Basis Matrix Multiplication (ABMM) is a practical Strassen-like algorithm that reduces the number of additions while preserving the same asymptotic exponent and admitting a provable O(n) component-wise error bound. However, optimized GPU support for ABMM remains underexplored. This paper presents BAG (Basis Alternative Matrix Multiplication on GPUs), a GPU-oriented implementation of ABMM for the NVIDIA Ampere architecture. A naive GPU implementation of ABMM suffers from excessive workspace and memory traffic, launch-dominated serialization across heterogeneous steps, and high sensitivity to recursion and blocking parameters. We design fused ABMM kernels to shrink workspace and eliminate redundant global-memory traffic, specialize register usage for different primitive patterns to expose intra/inter-kernel parallelism, and introduce a cost-model-based recursion policy together with a Roofline-guided blocking strategy to stabilize performance across problem sizes and shapes. On an NVIDIA A100, experimental results show that 1-level BAG reaches a break-even point against cuBLAS FP32 GEMM at dimension 1280 for square matrices, and recursive BAG achieves a 1.24 × speedup over cuBLAS FP32 GEMM at dimension 4096. Our code is available at https://doi.org/https://github.com/napleon-liu/bag.
Yao Liu, Ye-Wen Li, Zhonghai Zhang et al.· Proceedings of the Internati...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
