This work proposes a training-free model compression approach based on the tensor-train decomposition, whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS), and comprehensively investigates what language capabilities are preserved under training-free compression at different compression ratios.
Abstract
High-dimensional token embeddings underpin generative language models, as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which are particularly unaffordable for low-end devices. Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the tensor-train decomposition (TTD), whereby each pre-trained token embedding is converted into a lower-dimensional matrix product state (MPS). We investigate what language capabilities are preserved under training-free compression at different compression ratios, providing insights into the distinct redundancy structures captured by tensor-based versus pruning-based compression methods. We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e., Raspberry Pi). Our approach trades increased inference latency (no more than 0.5 ms/token reconstruction overhead on Raspberry Pi) for substantial memory and storage reduction, making it suitable for deployment scenarios where memory and storage are the primary bottleneck. Taking GPT family, OPT models, Qwen (2.5–0.5 B and 3–0.6 B) and MiniCPM4-0.5B as case studies, our approach for the embedding layer compression consistently achieves a compression factor 0.5× – 2.0×. The extension of our approach for dense layers (feed-forward layers and attention layers) compression, can improve the model language task performance in zero-shot reasoning tasks. Our performance analysis of different tasks reveals that tensor decomposition preserves higher-level logical reasoning capabilities (e.g., BoolQ, ARC-Challenge), while pruning-based methods like SliceGPT maintain advantages for tasks requiring broad lower-level lexical feature coverage (e.g., HellaSwag, WinoGrande), demonstrating that different compression approaches preserve complementary linguistic capabilities.
This survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability, and introduces a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup.
M. Tarasov, Salman Ahmadi-Asl, A. D. de Almeida et al.· 0 citations
Emergence of large scale protein language models (pLMs) has led to significant performance gains in predictive protein modeling. However, it comes at a high price of interpretability, and efforts to push representation learning toward explainable feature spaces remain scarce. The prevailing use of domain-agnostic and sparse encodings in such models fosters a perception that developing both parameter-efficient and generalizable models in a low-data regime is not feasible. In this work, we explore an alternative approach to develop compact models with interpretable embeddings while maintaining competitive performance. With the bidirectional long short-term memory autoencoder (BiLSTM-AE) model trained on positional property matrices, we introduce a soft weight matrix nondiagonality penalty and a one-hot encoded sequence clustering-based contrastive loss. As evidenced by Jacobian analysis, the penalty aligns embeddings with the initial feature space, whereas the contrastive loss organizes the latent space semantically. This combination leads to consistent improvements in performance on a suite of eight common peptide biological activity and physicochemical properties benchmarks. The use of amino acid physicochemical properties and density functional theory (DFT) derived cofactor interaction energies as input features provides a foundation for intrinsic interpretability, which we demonstrate on fundamental peptide properties. The resulting model is over 33,000 times more compact than the state-of-the-art pLM ProtT5. It demonstrates performance stability across diverse benchmarks without task-specific fine-tuning, showcasing that domain-tailored architectural design can yield highly parameter-efficient models with fast inference and preserved generalization capabilities.
Evgeniy Nam, Y. Din, Nikita S. Serov· Journal of Chemical Informat...· 0 citations
Support Vector Generation is introduced, a kernel-based framework that converts a frozen language model into an interpretable, training-free classifier for zero-and few-shot learning and suggests that SVG offers a viable path toward efficient, interpretable NLP systems under compute constraints.
Reduced Matrix Multiplication is proposed, a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights, and it is shown that the same principle extends to multimodal vision-language inference.
Language-model efficiency is often discussed through concepts that are related but not equivalent: token count, embedding dimensionality, parameter count, key-value cache size, and retrieved-context length. This article presents a structured critical review, supported by document-level traceability, of matrix and vector methods used to reduce storage and inference costs in language models and retrieval-augmented generation (RAG) systems. The analysis shows that matrix factorization, adaptive embeddings, and Matryoshka representations compress parameters or vectors but do not automatically decrease the number of input tokens. Direct token reduction requires sequence-level interventions, including appropriate tokenization, selection, prompt compression, gist tokens, and retrieved-context control. KV-cache compression reduces memory and latency without necessarily changing the tokens submitted to the model. We propose a mathematical framework that separates V (vocabulary size), d (embedding dimension), n (sequence length), k (number of retrieved documents or passages), and nKV (cache positions retained).
Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.
Shi-Cheng Wei· Theoretical and Natural Scie...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
