A training-free framework, BAL-ANCER, which achieves global budget allocation for mixed-precision quantization and low-rank correction through information-guided subspace matrices, allowing a principled greedy allocator to distribute compression bits and ranks across the entire model.
Training-free LLM compression avoids fine-tuning by approximating weight matrices from a small unlabelled calibration set; however, existing methods assign each layer an independent target rank with no global budget coordination, leaving parameter distribution across layers systematically suboptimal. We propose CLEAR (Convex-optimal Layer Energy Allocation and Representation), a training-free compression framework that combines an activation-whitened structured-dictionary representation with a provably optimal, globally coordinated budget allocation across layers. Ranks are distributed across all layers simultaneously via a convex water-filling optimization, whose KKT solution is provably optimal and achieves <10−3 pp precision on the dictionary-path parameter budget (the total realized retention ratio, including EoRA and outlier bypass parameters, deviates from the target by at most 0.02 pp in practice). Each layer is then compressed using activation-whitened structured sparse dictionaries (W≈DC, k-sparse columns), supplemented by an EoRA low-rank residual correction and an outlier input-channel bypass. Evaluated on five LLMs spanning 600 M to 8 B parameters across four model families, CLEAR outperforms the current state of the art on three of four evaluated models at compression ratio (CR) = 0.2. On LLaMA-3.2-1B, average accuracy across eight zero-shot benchmarks reaches 50.4% versus 42.7% for CoSpaDi and 37.6% for SVD-LLM, with perplexity reduced from 63.7 to 22.4; on LLaMA-3-8B, 65.5% versus 61.8% for CoSpaDi. Component ablations confirm +5.22 pp from sparse dictionaries and +1.12 pp from global rank allocation. KFAC Fisher covariance proves counterproductive in this setting, degrading accuracy by 8.1 pp due to numerical overflow in SiLU gating layers and consequent budget misallocation. CLEAR completes in 8–30 min on a single GPU, requires no gradient computation at any stage (forward-pass only, including the cascade activation refresh in Phase 3), and reduces peak deployment memory by approximately 20% at BFloat16 precision.
Large language models (LLMs) are often memory-bandwidth bound during autoregressive decoding, so reducing weight storage does not automatically produce parallel speedup when the compressed representation is irregular. We present SSQT, a post-training framework that jointly applies hardware-aligned structured sparsification, sensitivity-driven outlier preservation, and sparse-aligned low-bit group quantization. SSQT estimates parameter importance from calibration data with a diagonal empirical-Fisher approximation, avoiding construction of the full Hessian, and, in the default 4-bit configuration, stores fewer than 1% high-sensitivity weights on a separate FP16 residual path. The remaining weights are packed in regular attention blocks and contiguous feed-forward-network channel groups; quantization metadata is secondarily quantized and decoded inside the matrix-multiplication tile rather than by globally expanding the model to FP16. Experiments on Llama 2 and Falcon models report task quality, calibration cost, packed storage, latency regularity, cross-GPU results, and hardware counters. In the default 4-bit configuration, SSQT uses 24.3% of the FP16 model-memory footprint on Llama 2-13B, keeps the relative WikiText2 perplexity increase at 4.8%, and improves the per-sequence decoding rate by up to 2.33 × on the A100 tensor-parallel configuration. On Llama 2-13B, Tensor Core utilization rises from 28.7% to 62.4%, memory-bandwidth utilization falls from 92.3% to 41.8%, and pipeline stalls fall from 34.6% to 11.0%. These results show that compression and execution layout must be co-designed to convert model-size reduction into practical parallel inference acceleration.
We revisit Kashin-decomposition-based weight quantization for large language models and propose an improved algorithm with stronger convergence properties and structured, efficient orthogonal transforms. The method retains the core factorization of each weight into two components -- one with bounded infinity norm and the other with bounded infinity norm after an orthogonal transformation -- but replaces the dense random orthogonal matrix with a sign-randomized Discrete Cosine Transform (DCT), reducing the per-iteration cost from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$. The proposed greedy algorithm with alternating updates guarantees the four-peak distribution required for stable 2-bit clustering of each factor and admits closed-form initialization of cluster centers, removing the multi-restart k-means bottleneck of prior work. Composed with OPTQ-style sequential error compensation and QuIP-style incoherence preprocessing, the resulting JAX pipeline is competitive with OPTQ, QuIP, QuIP-RG and a fine-tuning- and vector-quantization-free variant of QuIP# at 4-bit per channel on OPT, Llama-2 and Pythia, with favorable wall-clock scaling. The bounded-$\ell_\infty$ factorization is also notably robust: on stress configurations where QuIP variants diverge to four-digit perplexity (Pythia-6.9B) or abort with NaNs in LDL back-substitution (Mistral-7B), Kashin-DCT remains numerically stable and stays close to FP16 baseline. At inference time, each weight decomposes into two 2-bit factor codes per channel that are structurally suited to native-2-bit hardware.
Daria Cherniuk, A. Rudikov, B. Kashin et al.· 0 citations
Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the"Compression Trinity,"a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank"lazy"adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.
Across five models from 1B to 32B, SoftWater outperforms the released WaterSIC quantizer at matched head rates on 59 of 60 test points, using none of that pipeline's refinements and cutting head-induced KL by $6.5\times$--$8.3\times at 2 bits.
This work identifies and formalizes the principle that organizes the Great Inversion, the Great Inversion: allocation-flexible coding rewards energy concentration, whereas the grouped shared-scale quantization a deployed matrix instruction performs rewards within-group flattening.
E. Jokar· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
