It is shown that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision, and a deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods.
Abstract
Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.
Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.
Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.
PoLoRA is introduced, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates.
Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower· arXiv.org· 1 citation
Parameter-efficient fine-tuning (PEFT) is a key technique for adapting large pre-trained language models to downstream tasks with minimal parameter updates. However, existing PEFT methods often suffer from slow convergence, gradient noise, and weak alignment between learned features and task semantics. To this end, we develop a low-rank adaptation framework based on gradient spectral alignment. Specifically, we (1) perform spectral decomposition of the pre-trained gradient covariance matrix and apply eigenvalue scaling to retain 95% of the spectral energy, optimizing parameter initialization; (2) design a dual gradient projector that combines orthogonal and spherical projections to suppress gradient noise, reducing its variance to 38.2% of standard LoRA; and (3) introduce a dynamic eigenvalue scaling mechanism that adaptively recalibrates principal component weights via a nonlinear scaling function, enhancing representational capacity. Experiments on the GLUE benchmark show that our method achieves performance comparable to full fine-tuning while training only 0.1% of parameters, outperforming AdaLoRA and DoRA by 6.2% and 5.3%, respectively. Additionally, our approach improves the mathematical reasoning accuracy of LLaMA-7B on GSM8K by 7.8%. This work provides novel insights into improving the efficiency, stability, and adaptability of PEFT frameworks. The code is available at https://github.com/rainylover/GSLoRA.
This work proposes a curvature-conditioned multiscale momentum method with sphere constraints, which significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters).
Shuchen Zhu, Yu-Xin Fang, Mingze Wang et al.· 0 citations
This work intervenes on one factor at a time inside a fixed pipeline, holding candidate scoring and voting constant while they vary the search dimension, the subspace that carries the perturbation, and its norm.
Taeyeon Kim, Ahhyun Kim, Taehyeon Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
