Aug 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP, pp. 1-17· 0 citations
Medicine
TL;DR
This work unify these approaches under a Triplet Matrix Decomposition framework and reveals that frequency-domain methods can surpass low-rank approaches when optimal frequency components are selected, and this advantage stems from orthogonal transformation matrices and flexible basis vector combinations.
Abstract
Parameter-Efficient Fine-Tuning (PEFT) methods enable adapting large pre-trained models to downstream tasks with minimal overhead. Current approaches predominantly rely on low-rank decomposition to reparameterize weight increment matrices, assuming that model updates follow low-rank patterns. However, weight updates during fine-tuning may exhibit more complex statistical properties. An alternative paradigm decomposes weight increment matrices into frequency-domain components, offering potentially superior expressivity through flexible frequency component combinations. In this work, we unify these approaches under a Triplet Matrix Decomposition framework and rigorously compare their expressivities. Our analysis reveals that frequency-domain methods can surpass low-rank approaches when optimal frequency components are selected, and this advantage stems from orthogonal transformation matrices and flexible basis vector combinations. Building on these insights, we propose Learnable Orthogonal Adaptation (LoTA), a novel PEFT method that learns task-specific transformations and adaptive basis combinations. LoTA employs cascaded Householder transformations to construct orthogonal matrices with minimal parameters while ensuring exploration of the complete orthogonal space. We address the discrete optimization challenge of basis selection through finite-difference gradient approximation, enabling end-to-end backpropagation. Extensive experiments including natural language understanding, mathematical reasoning, commonsense reasoning, computer vision and visual instruction tuning tasks demonstrate that LoTA achieves superior parameter efficiency and performance compared to existing PEFT methods.
In this paper, we propose AdaMSS, an adaptive multi-subspace approach for parameter-efficient fine-tuning of large models. Unlike traditional parameter-efficient fine-tuning methods that operate within a large single subspace of the network weights, AdaMSS leverages subspace segmentation to obtain multiple smaller subspaces and adaptively reduces the number of trainable parameters during training, ultimately updating only those associated with a small subset of subspaces most relevant to the target downstream task. By using the lowest-rank representation, AdaMSS achieves more compact expressiveness and finer tuning of the model parameters. Theoretical analyses demonstrate that AdaMSS has better generalization guarantee than LoRA, PiSSA, and other single-subspace low-rank-based methods. Extensive experiments across image classification, natural language understanding, and natural language generation tasks show that AdaMSS achieves comparable performance to full fine-tuning and outperforms other parameter-efficient fine-tuning methods in most cases, all while requiring fewer trainable parameters. Notably, on the ViT-Large model, AdaMSS achieves 4.7% higher average accuracy than LoRA across seven tasks, using just 15.4% of the trainable parameters. On RoBERTa-Large , AdaMSS outperforms PiSSA by 7% in average accuracy across six tasks while reducing the number of trainable parameters by approximately 94.4%. These results demonstrate the effectiveness of AdaMSS in parameter-efficient fine-tuning. The code for AdaMSS is available at https: //github.com/jzheng20/AdaMSS .
Jingjing Zheng, Wanglong Lu, Yiming Dong et al.· Neural Information Processin...· 0 citations
Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support-level design does not explicitly select sparse, structured regions that are task-salient for downstream adaptation. To address this limitation, we propose SPIRA, a PEFT framework that separates support discovery from parameter-efficient adaptation. During a short warm-up, the Relative Information-Geometric Potential (RIGP) identifies a high-saliency sparse seed support from a squared-gradient base statistic. This online criterion draws on the local-sensitivity perspective of information geometry without constructing the full Fisher information matrix. The selected seed support determines active input and output indices, which in turn define a structural closure. SPIRA fixes this closure as the adaptation mask and trains active-axis low-rank factors whose parameter count scales with the active dimensions. Experiments across computer vision, natural language processing, and vision-language modeling benchmarks show that SPIRA remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhongyi Wen, Zhikai Zhai, Guomin Sun et al.· IEEE Transactions on Pattern...· 0 citations
As pretrained language models grow, full parameter updates become prohibitively expensive, motivating parameter-efficient fine-tuning (PEFT). LoRA decomposes weight updates into low-rank factors, but its per-module cost $r(m{+}n)$ imposes a structural floor at rank one. FourierFT breaks this floor via frequency-domain coefficients, yet requires a dense inverse transform at each forward step. We propose LYRA (Low-frequencY Rank Adaptation), which parameterizes each weight update with a small set of low-frequency 2D discrete cosine transform coefficients, chosen separately along each axis and contiguous in its canonical form. This separable structure admits a factored forward pass of three small matrix multiplications, avoiding dense reconstruction entirely. On GLUE and SuperGLUE with the BERT-base and RoBERTa-base encoders, LYRA matches FourierFT at identical parameter and optimizer-memory budgets while achieving the lowest peak GPU memory among all methods tested.
Sayed Muhsin, Seok-Bum Ko· IEEE Signal Processing Lette...· 0 citations
The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Harshavardhan Adepu, Li Zhang, Sanjiv Kumar et al.· 0 citations
Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.
Parameter-efficient fine-tuning (PEFT) is a key technique for adapting large pre-trained language models to downstream tasks with minimal parameter updates. However, existing PEFT methods often suffer from slow convergence, gradient noise, and weak alignment between learned features and task semantics. To this end, we develop a low-rank adaptation framework based on gradient spectral alignment. Specifically, we (1) perform spectral decomposition of the pre-trained gradient covariance matrix and apply eigenvalue scaling to retain 95% of the spectral energy, optimizing parameter initialization; (2) design a dual gradient projector that combines orthogonal and spherical projections to suppress gradient noise, reducing its variance to 38.2% of standard LoRA; and (3) introduce a dynamic eigenvalue scaling mechanism that adaptively recalibrates principal component weights via a nonlinear scaling function, enhancing representational capacity. Experiments on the GLUE benchmark show that our method achieves performance comparable to full fine-tuning while training only 0.1% of parameters, outperforming AdaLoRA and DoRA by 6.2% and 5.3%, respectively. Additionally, our approach improves the mathematical reasoning accuracy of LLaMA-7B on GSM8K by 7.8%. This work provides novel insights into improving the efficiency, stability, and adaptability of PEFT frameworks. The code is available at https://github.com/rainylover/GSLoRA.
