FluxBin is proposed, an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel and introduces Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency.
Abstract
While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (\textbf{F}lexible \textbf{L}UT-based \textbf{U}ltra-low-bit e\textbf{X}ecution with \textbf{Bin}ary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at https://github.com/nicyyyy/FluxBin.
This work presents the FQTree algorithm for fine-grained quantization-aware training of BDTs, together with the QXGB framework for automatic hardware generation, and applies this quantization during boosting so that later trees adapt to the errors of the already-quantized ensemble.
Zhiqiang Que, Chang Sun, Haiyan Wang et al.· 0 citations
The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding
Zhigeng Liu, Zhiyuan Ning, Ruixiao Li et al.· 0 citations
PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget, shows that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.
Transformers outperform traditional neural networks but face high computational and memory costs, limiting edge device deployment. Although many hardware accelerators aim to address this, the original Transformer structure still restricts the optimization effect. A recent breakthrough, mixture-of-depths (MoDs), employs conditional computation and effectively reduces the computational complexity of large language models, providing a valuable opportunity for designing an efficient hardware accelerator. However, when applied to vision transformers, MoD suffers from accuracy degradation and excessive external memory access (EMA). Therefore, this article presents ME-MoD, the first memory-efficient MoD-based vision transformer inference accelerator, leveraging the idea of reordering and algorithm-hardware-dataflow codesign. Algorithmically, distribution adjustment forward (DAF) and routing decision forward (RDF) techniques restore accuracy and alleviate memory access costs through token reordering. Architecturally, a LayerNorm-Routing (L-R) fusion module and a token reordering and sequential recording module enhance computational efficiency while minimizing memory overhead. In addition, a token-stationary layer fusion dataflow and an on-chip dynamic memory module are designed, which further optimizes the EMA caused by the intermediate results of interlayer computation of valid tokens routed by MoD. With negligible accuracy loss, our ME-MoD accelerator achieves $1.62\times $ inference speed up, eliminates 46.5% of the external memory bandwidth requirement and 45.2% of energy consumption compared with standard MoD. It achieves 23.6 TOPS/W energy efficiency, which is $4.02\times $ improvements compared with state-of-the-art (SOTA) designs.
Jia-Ning Chen, Wen-Long Ma, Yun-Chuan Li et al.· IEEE Transactions on Very La...· 0 citations
TERINT-GEMV is designed, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping.
Tomas Sousa Pereira, Ruibao Wang, Ikenna Nwozo et al.· International Symposium on L...· 0 citations
AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator that removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss across LLMs from 3B to 70B.
Junyi Luo, Xin Jiang, Tai-Hao Wen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
