Preprint

UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge

Tianhao Jiang Hang Gu Teng Wang Qian-Yu Cheng Zhen-Dong Zheng Cheng Tang Qiyue Su Wenqi Lou Lei Gong Chao Wang Xi Li Xue-Hai Zhou

Aug 2026 · 0 citations · 46 references

Computer Science

TL;DR

Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively.

Abstract

Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

FlashQuant: Sparse-Dense Fusion for Memory-Efficient Outlier-Aware LLM Inference

FlashQuant fuses the dense GEMM and sparse outlier SpMM paths into a single GPU kernel, enabling on-chip reuse of activation and output tiles across heterogeneous computations, and introduces three key techniques: sparse-dense tiling, which aligns outlier processing with dense GEMM tiles, and pipelined scheduling, whic...

Jun-Qing Lin, Jing-Wei Sun, Zhengding Hu et al. · 0 citations

#artificial intelligence Preprint Sep 2026

Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

This work presents an end-to-end hardware-software co-design framework that compresses expert weights into hardware-native, low-precision sparse representations and accelerates their execution on SpTCs, establishing hardware-software co-design as a practical path toward scalable and efficient MoE deployment.

Kwanhee Lee, Namhoon Lee, Dan Alistarh · 0 citations

#artificial intelligence Preprint Oct 2026

Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression

SVD-based pruning and quantization have recently emerged as a promising strategy for the ultra-efficient compression of large language models. In these methods, compression is performed in two stages: components are first truncated, and the remaining ones are subsequently quantized. Although this decoupled pipeline ben...

Hankyul Kang, Jongbin Ryu · 0 citations

#artificial intelligence Preprint Aug 2026

QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization

This work presents QTEA, a sub-2-bit PTQ framework that quantizes weights into ternary values and uses salient weights as residual error compensators and develops a lookup-table based kernel that achieves 7.2% faster per-token generation over an FP16 baseline.

Yi-Pin Guo, Arun George, Jie Fu et al. · 1 citation · ⚡1

#machine learning Preprint Oct 2026

PageWeaver: KV-Guided Query Unions for Sparse Attention

Dynamic sparse attention limits the KV pages selected by each query, but a small support does not necessarily yield efficient GPU work. Query unions share page loads and populate Tensor Core tiles; their cost depends on which queries are grouped together. We present PageWeaver, an execution design that uses selected-pa...

Zhi-Yuan Li, Zi-Han Li, Ze-Fang Yuan et al. · 1 citation

Preprint Aug 2026

Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the"Compression Trini...

Mohammad Mozaffari · 1 citation