Skip to content

ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs

Sep 2026 · arXiv.org · Vol abs/2609.36120 · 0 citations · 33 references
Computer Science

TL;DR

ThinQuant is introduced, a data selection procedure which reduces the required number of calibration data points and an exact reduction of the associated optimization on this reduced calibration set using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant.

Abstract

Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an $r$-dimensional subspace with $r<d$, making optimization over a $d\times d$ rotation equivalent to optimizing a $d\times r$ matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requi...

Mahsa Salmani, Sayeh Sharify, Hesham Mostafa · 0 citations
Sep 2026

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

Smaller activation outliers do not necessarily imply better low-bit quantization: their alignment with the quantizer matters. We introduce PrismQuant, a quantizer-aware rotation framework that aligns the leading activation eigenspace with the constant group subspace of asymmetric grouped INT4. The affine offsets repres...

Yan-Long Chen, Yi-Ning Chen, Song Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

A Unified Hessian-Guided Framework for Post-Training Quantization

Low-bit quantization enables efficient deployment of large language models by reducing memory footprint and computational cost. Achieving high accuracy at low precision requires selecting quantization ranges, linear transformations, and weight rounding decisions. Existing approaches typically optimize these components...

M. Toftrup, Marco Federici, Boris van Breugel et al. · 0 citations

SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models

SPHQuant is a rotation-free spherical weight-only quantization framework for VLMs that isolates outlier magnitude into the radius while keeping directions bounded and statistically regular and designs a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radia...

Ke-Wei Zhang, Zheng Chen, Hao-Tong Qin et al. · 1 citation
#artificial intelligence Preprint Sep 2026

JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

JustQuant is proposed, a simple yet effective framework that moves the complexity of low-bit quantization from deployment-time operators into the training process and introduces Theseus QAD, a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement...

Kai-Cheng Yang, Kai-Sen Yang, Chun-Yu Liu et al. · 0 citations
Preprint Oct 2026

Quantization of Large Models with Weight Outliers and Sparsity

Post-training quantization (PTQ) reduces the memory burden of large model deployment by casting the high precision trained weights to fewer bits. While this promotes memory-efficient inference, it also incurs quantization error that impairs model performance. To compensate for quantization error, the present contributi...

Leonidas Tsigkounakis, Abraham Jaeger Mountain, Yilang Zhang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.