2026· Annual Meeting of the Association for Computational Linguistics· pp. 35255-35273· 0 citations· 83 references
Computer Science
TL;DR
A novel Le arnable Lo w-R ank A daptation (LeLoRA) framework that utilizes dynamically learned fine-tuning strategies to facilitate the effective adaptation of LLMs and provides compelling evidence that LeLoRA consistently outperforms existing baselines in adapting LLMs.
Abstract
Fine-tuning large language models (LLMs) is an effective approach to enhancing their performance on specialized downstream tasks. Among the various techniques, low-rank adaptation has garnered significant attention due to its ability to maintain the full performance of fine-tuning while enhancing computational efficiency. However, existing approaches often rely on manually specified and fixed hyperparameters to identify the trainable components within weight matrices, resulting in suboptimal performance and low parameter efficiency. This paper presents a novel Le arnable Lo w-R ank A daptation (LeLoRA) framework that utilizes dynamically learned fine-tuning strategies to facilitate the effective adaptation of LLMs. Our framework integrates an LLM with a policy network that automatically and adaptively generates matrix-specific adaptation strategies to identify the trainable components of each weight matrix, taking into account their unique characteristics, such as singular values and matrix norms. A reinforcement learning-based optimization algorithm is then employed to iteratively update the LLM and the policy network, ensuring that the generated strategies adapt in real time to the evolving states of the LLM. Extensive experiments have been conducted across various natural language processing tasks. The results across ten different LLMs, ranging from 125M to 70B parameters, provide compelling evidence that LeLoRA consistently outperforms existing baselines in adapting LLMs.
Parameter-efficient fine-tuning (PEFT) methods adapt foundation models to specific domains by selectively updating only a small subset of critical parameters, significantly reducing computational costs. Among these methods, Sparse Low-Rank Adaptation (SoRA) has emerged as an effective approach for fine-tuning large language models. This paper investigates SoRA algorithm's performance in large-scale models, particularly analyzing learning dynamics and scaling effects in wide neural networks. We find that employing a uniform learning rate across all parameter matrices limits learning efficiency. To address this, we derive an improved learning rate configuration and propose SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally. This modification aligns with theoretical optima and mitigates the suboptimality of the original SoRA method. Experiments show that SoRA+ improves fine-tuning performance by up to 2% while accelerating convergence.
Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.
Shi-Cheng Wei· Theoretical and Natural Scie...· 0 citations
Large language models have demonstrated outstanding performance in the field of natural language processing, but pretrained models are difficult to adapt to specific tasks. Fine-tuning technology has become a key solution, but its application is limited by problems such as complex parameter selection, high training costs, and uncertain effect evaluation. This study aims to construct a complete framework for optimizing fine-tuning parameters of large language models, to achieve accurate prediction and scientific evaluation of fine-tuning effects, and to solve the problems of overfitting and high-performance adaptation under limited data. By constructing a multi-task mixed dataset, comparative experiments were designed using the controlled variable method. The influence of key hyperparameters such as learning rate and batch size was systematically analyzed, the performance of classic and modern optimization algorithms was compared, and a prediction mechanism was established by combining regression and classification models. A comprehensive evaluation system based on "performance-efficiency-resource" was constructed. The results show that a learning rate of 1E-3 and a batch size of 64 are the optimal configurations. Modern optimization algorithms converge faster, and multinomial regression and neural networks perform best in effect prediction. The performance of small-scale models after fine-tuning can surpass that of un-fine-tuned large models, significantly reducing computational costs. This study provides a scientific basis for optimizing fine-tuning parameters, improves resource utilization efficiency, and promotes the efficient implementation of large language models in vertical fields.
Yi-He Ding· International Conference on...· 0 citations
This work proposes eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework grounded in gradient boosting theory that provides theoretical analysis establishing convergence guarantees and expressiveness bounds, which formally justify why weaker (lower-rank) adapters, when properly combined, can match or exceed the performance of stronger ones.
Yifei Zhang, Hao Zhu, Haoran Shi et al.· Proceedings of the 32nd ACM...· 0 citations
Parameter-efficient fine-tuning (PEFT) methods have become essential for adapting large language models (LLMs) to downstream tasks without updating all parameters. However, the optimal PEFT strategy varies significantly across model architectures, task types, and resource constraints, making method selection a non-trivial problem. This paper presents PEFT-Assess, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability. We benchmark these methods on LLaMA-7B and Qwen-7B across six NLP tasks from the GLUE and SuperGLUE benchmarks. Furthermore, we propose a gradient boosting-based strategy selector that predicts the optimal PEFT method given task characteristics and resource budgets. Experiments show that LoRA achieves the best overall adaptability score of 0.847, while our strategy selector attains 91.2% prediction accuracy for recommending the optimal PEFT method.
Zhaoyang Hu, Shixuan Cao, Yuqi Sun et al.· International Conference on...· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
