The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Abstract
Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· IEEE Transactions on Emergin...· 0 citations
As pretrained language models grow, full parameter updates become prohibitively expensive, motivating parameter-efficient fine-tuning (PEFT). LoRA decomposes weight updates into low-rank factors, but its per-module cost $r(m{+}n)$ imposes a structural floor at rank one. FourierFT breaks this floor via frequency-domain coefficients, yet requires a dense inverse transform at each forward step. We propose LYRA (Low-frequencY Rank Adaptation), which parameterizes each weight update with a small set of low-frequency 2D discrete cosine transform coefficients, chosen separately along each axis and contiguous in its canonical form. This separable structure admits a factored forward pass of three small matrix multiplications, avoiding dense reconstruction entirely. On GLUE and SuperGLUE with the BERT-base and RoBERTa-base encoders, LYRA matches FourierFT at identical parameter and optimizer-memory budgets while achieving the lowest peak GPU memory among all methods tested.
Sayed Muhsin, Seok-Bum Ko· IEEE Signal Processing Lette...· 0 citations
Circuit Fine-Tuning is introduced, a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training to isolate the response of the backbone to the target distribution rather than the preferences of a particular classifier.
Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support-level design does not explicitly select sparse, structured regions that are task-salient for downstream adaptation. To address this limitation, we propose SPIRA, a PEFT framework that separates support discovery from parameter-efficient adaptation. During a short warm-up, the Relative Information-Geometric Potential (RIGP) identifies a high-saliency sparse seed support from a squared-gradient base statistic. This online criterion draws on the local-sensitivity perspective of information geometry without constructing the full Fisher information matrix. The selected seed support determines active input and output indices, which in turn define a structural closure. SPIRA fixes this closure as the adaptation mask and trains active-axis low-rank factors whose parameter count scales with the active dimensions. Experiments across computer vision, natural language processing, and vision-language modeling benchmarks show that SPIRA remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhongyi Wen, Zhikai Zhai, Guomin Sun et al.· IEEE Transactions on Pattern...· 0 citations
Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed to remedy such performance degradation by defining the state-of-the-art low-rank adaptation. Instead of being constrained by the conventional formulation, this approach uses the minimal possible rank-8 adapter, full linear module coverage, and a learned learning rate. Extensive ablations offer two important discoveries that structural module coverage has more impact than mere adapter rank, and high learning rate is indispensable to provide enough convergence with the limited number of parameters. We show in the experiment that this configuration manages to restore the model performance to the 93.73% and 90.18% on Banking77 and CLINC150 respectively. S1 configuration attains the baseline parity, while only updating 1.56%-1.60% total trainable parameters with a maximal 2.96GB memory. This proves that it is possible to train a high-accuracy transformer on the hardware targeted at consumers, for example the NVIDIA RTX 5060 Ti.
Harsh Anand, Sofia Singh, Rahul Agrawal et al.· 2026 International Conferenc...· 0 citations
Parameter-efficient fine-tuning (PEFT) methods adapt foundation models to specific domains by selectively updating only a small subset of critical parameters, significantly reducing computational costs. Among these methods, Sparse Low-Rank Adaptation (SoRA) has emerged as an effective approach for fine-tuning large language models. This paper investigates SoRA algorithm's performance in large-scale models, particularly analyzing learning dynamics and scaling effects in wide neural networks. We find that employing a uniform learning rate across all parameter matrices limits learning efficiency. To address this, we derive an improved learning rate configuration and propose SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally. This modification aligns with theoretical optima and mitigates the suboptimality of the original SoRA method. Experiments show that SoRA+ improves fine-tuning performance by up to 2% while accelerating convergence.
