Skip to content

RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models

Sep 2026 · 0 citations · 52 references
Computer Science

TL;DR

Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions.

Abstract

Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ). However, most PTQ methods are designed for unimodal large language models (LLMs). These methods treat quantization errors as isotropic perturbations under the Euclidean assumption, which provides weak guidance on directions most sensitive to quantization in VLMs. Consequently, directly adapting unimodal PTQ approaches or solely employing modality-specific scaling often leads to uneven bit-width distribution and inconsistent performance in low-bit settings. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric. RGSQ identifies modality-specific sensitive directions via Riemannian manifold mappings built from modality-partitioned empirical Fisher factors and fused into a modality-aware Kronecker-structured metric. We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes. Finally, we apply a whitening transformation that maps the Riemannian objective to an equivalent Euclidean form, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions. Across an extensive and diverse set of mainstream VLM benchmarks, RGSQ achieves the highest accuracy and stability under extremely low-bit settings (W2A8 and W3A8). It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9% and surpasses single-modality improvements by up to 8.6%.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding

Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V|, while hyperbolic curvature helps selectively, and characterize boundary collapse in naive hyperbolic recurrence and show how Mobius stabilization restores trainability.

Fang Li · 0 citations

Beyond Reconstruction Loss in Post-Training Quantization: Balanced Fitting for Large Vision-Language Models

Experiments on multiple LVLMs show that the Balanced Fitting method consistently outperforms prior PTQ approaches under both weight-only and weight-activation quantization, while lower reconstruction loss does not reliably translate into better downstream performance.

Min-Chan Kang, Kyeonghye Park, Seungyeon Sa et al. · 0 citations

SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models

SPHQuant is a rotation-free spherical weight-only quantization framework for VLMs that isolates outlier magnitude into the radius while keeping directions bounded and statistically regular and designs a hardware-friendly GEMV kernel that keeps the direction codebook small enough for shared-memory lookup and packs radia...

Ke-Wei Zhang, Zheng Chen, Hao-Tong Qin et al. · 1 citation
Sep 2026

SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization

Post-training quantization reduces the deployment cost of vision-language models (VLMs), but preserving multimodal capabilities at low bit widths remains challenging. Existing methods rely on modality- or token-level gradient statistics, which are susceptible to cross-sample variations in visual-to-textual token ratios...

Zhen-Hao Shang, Hai-Zhao Jing, Hao-Kui Zhang et al. · 0 citations
Sep 2026

Technical note on: Zero-Training Feature-Space Alignment via Information Geometry

Zero-Training Fisher Geometry Alignment (ZFGA), a closed-form method that improves robustness under covariate shift without modifying model parameters, is proposed, offering a lightweight and deterministic alternative to optimization-based test-time adaptation.

Behraj Khan, T. Syed, Syed Ahmad Chan Bukhari · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.