Aug 2026· International Symposium on Low Power Electronics and Design· 0 citations· 24 references
Computer Science
TL;DR
TERINT-GEMV is designed, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping.
Abstract
Emerging 1.58-bit large language models such as BitNet restrict weights to {-1, 0, +1}, dramatically reducing model size. Yet commodity hardware fails to fully exploit this structure: each ternary value carries only log2 (3) ≈ 1.58 bits of information but must occupy a 2-bit container, wasting 21% of memory bandwidth. Furthermore the lack of native mixed-precision support forces costly runtime dequantization. We present a hardware-software co-design that addresses both inefficiencies. On the software side, we compress ternary weights to 1.5 bits per weight via a 64-entry codebook with 6-bit indices, sacrificing only 5.4% of representational capacity; Vector Quantization aware fine-tuning retains 92.6% of baseline accuracy across six benchmarks. On the hardware side, we design TERINT-GEMV, a multiplier-free bit-serial accelerator that ingests the 6-bit indices directly through an on-chip lookup table, computing ternary × INT8 dot products with dynamic zero-skipping. Synthesized using a 28 nm commercial PDK, TERINT-GEMV achieves 99.44 TOPS/W, outperforming recently published BitNet accelerators by up to 7.2× in energy efficiency, and delivers 0.065 J/token with a 2.1× throughput gain over the leading CPU-based ternary runtime at one fifth the power.
Experimental results demonstrate that L2Mersit achieves the highest accuracy among all 6-bit exponent-encoded formats while reducing the hardware complexity of auxiliary units for low-precision computing, resulting in a 62.7% area reduction.
M. Kim, Hyeonseong Kim, Ik-Joon Chang et al.· International Symposium on L...· 0 citations
AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator that removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss across LLMs from 3B to 70B.
Junyi Luo, Xin Jiang, Tai-Hao Wen et al.· 0 citations
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which pose significant challenges to low-bit quantization. In this work, we investigate sparse and low-bit activation strategies tailored for 1-bit LLMs. We first introduce BitNet v2 a4.8, a hybrid quantization-sparsification architecture designed to mitigate quantization errors arising from outlier activation channels. BitNet v2 a4.8 utilizes 4-bit quantization for inputs to attention and feed-forward layers, while sparsifying intermediate activations and subsequently quantizing them to 8 bits. To further leverage the 4-bit computation capabilities of next-generation GPUs, we present BitNet v2, a novel framework that enables fully 4-bit activations for 1-bit LLMs. To address the outlier activation channels, we propose $/mathcal{H}$, a lightweight module that applies an online Hadamard transformation prior to quantization, effectively regularizing activation distributions into more Gaussian-like forms conducive to low-bit representation. Extensive experiments demonstrate that BitNet v2 a4.8 achieves performance comparable to BitNet b1.58 with similar training costs. Furthermore, BitNet v2, when trained from scratch with 8-bit activations, matches the performance of BitNet b1.58. Remarkably, training BitNet v2 directly with native 4-bit activations leads to only marginal performance degradation, while significantly reducing computational overhead during batched inference.
Hongyu Wang, Shuming Ma, F. Wei· IEEE Transactions on Pattern...· 0 citations
Large language models (LLMs) are increasingly constrained by memory capacity, weight bandwidth, and checkpoint storage during deployment. Existing low-bit compression methods mainly follow two directions. Scalar or group-wise quantization is simple and compatible with efficient low-precision kernels, but its representation capacity becomes limited when the target budget approaches 2 bits per weight. Vector-quantized weight compression provides a richer block-level representation, but usually introduces explicit codebooks, index lookup, and additional storage accounting. This paper presents BiSCo-LLM, a codebook-free binary spherical coding framework for extreme low-bit LLM weight compression. The core pipeline is built on three components. First, local weight chunks are mapped onto a unit hypersphere and binarized into compact spherical codes, so that the main payload is a bit-packed sign stream rather than explicit VQ centroids. Second, a residual BSQ stage encodes the reconstruction error left by the base spherical codec, providing an explicit rate-distortion path without stored codebooks. Third, category-wise recovery distillation is performed after replacing each Transformer module category, reducing the mismatch between local weight reconstruction and assembled model behavior. A small 8-bit protected-channel path is used as an auxiliary stabilization mechanism for sensitive channels and is counted separately from the BSQ payload. The reported storage budget includes binary codes, neural decoders, protected-channel payloads, LoRA adapters, and metadata.
Yuantian Shao, Peisong Wang, Zhilei Liu et al.· 0 citations
T-ACE is presented, a Ternary Accuracy-aware Compute Engine that enables efficient ternary LLM inference under PTQ by jointly designing the data representation and execution pipeline and integrates on-the-fly decoding and scaling directly into the ternary GEMM pipeline.
Wonseok Jung, Junseok Kang, Sangwon Shin et al.· International Conference on...· 0 citations
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted"A"on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.
AI OneBit, A. Malik, Poojith et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
