Reduced Matrix Multiplication is proposed, a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights, and it is shown that the same principle extends to multimodal vision-language inference.
Abstract
Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications. We propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off. Across language models ranging from 1B to 70B parameters, we find that reduction tolerance depends on the model family, task, component, and retention ratio, although it often improves with model scale. Under moderate reduction, RMM remains robust across the evaluated discriminative, autoregressive generation, and long-context settings. We further show that the same principle extends to multimodal vision-language inference. Mechanistic ablations reveal a structural asymmetry within Transformers: attention-side computations are substantially more reducible than MLP components. Finally, wall-clock benchmarks with custom kernels on an NVIDIA A100 show that these computational savings can translate into practical runtime gains, especially at longer sequence lengths. Together, these results position RMM as a scalable direction for input-adaptive inference-time optimization.
COEC (Calibrated Orthogonal-Equivalence Compensation), a training-free compensation framework that applies alternating left and right orthogonal rotations to the retained weight, improves perplexity on every model and zero-shot accuracy in most settings over existing compensation methods.
This work unify these approaches under a Triplet Matrix Decomposition framework and reveals that frequency-domain methods can surpass low-rank approaches when optimal frequency components are selected, and this advantage stems from orthogonal transformation matrices and flexible basis vector combinations.
Zhekai Du, Yinjie Min, Dan Zhang et al.· IEEE Transactions on Pattern...· 0 citations
REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· IEEE Transactions on Emergin...· 0 citations
PoLoRA is introduced, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates.
Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower· arXiv.org· 1 citation
Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.
Shi-Cheng Wei· Theoretical and Natural Scie...· 0 citations
This work proposes a training-free model compression approach based on the tensor-train decomposition, whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS), and comprehensively investigates what language capabilities are preserved under training-free compression at different compression ratios.
Mingxue Xu, Y. Xu, Danilo P. Mandic· Machine Intelligence Researc...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduSep 2, 2026
A new method, called CW-Net, translates the reasoning process of an autonomous vehicle’s AI system into understandable concepts that explain its behavior.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
