Skip to content

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Sep 2026 · 0 citations · 18 references
Computer Science

TL;DR

This work regularizes the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones, and proposes a principled rank-allocation method based on classical sparsity-inducing technique in signal processing and statistics.

Abstract

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0<p<1$, which is a classical sparsity-inducing technique in signal processing and statistics. Specifically, we regularize the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones. We derive the corresponding proximal subproblem and reduce the matrix optimization to a two-dimensional problem, leading to an implicit thresholding criterion for identifying redundant components. Experiments on natural language understanding and question-answering tasks demonstrate that the proposed method achieves competitive performance with existing LoRA baselines.

View source

Similar papers

Review Open access Aug 2026

Low-Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Language Models

This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them, and compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation...

Shi-Cheng Wei · 0 citations
Aug 2026

Efficient sparse low-rank adaptation for large language models.

This paper investigates SoRA algorithm's performance in large-scale models, and derives an improved learning rate configuration and proposes SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally and mitigates the suboptimality of the original SoRA method.

Jun-Ren Qiao, Shi-Song Wu, Wen-Qing Su · 0 citations
Conference Open access Sep 2026

SeMi-LoRA: Enhancing Low-Rank Adaptation via Separation and Mixing

This work proposes SeMi-LoRA (Separation and Mixing LoRA), a novel framework that enables complete high-rank updates while preserving mergeability and consistently outperforms strong PEFT baselines with favorable parameter efficiency.

Zhen-Fei Yang, Beiming Yu, Peiqin Lin et al. · 0 citations

Stabilizing the Dynamic Low-Rank Training

The gradient flow of the best rank-$r$ approximation is derived and it is pointed out that the offset of DLRT comes from a curvature-coupling term which is large and thus non-negligible under aggressive compression, and a stable dynamic low-rank training method is proposed, named SDLRT.

Zhong-Han Xu, Ling Wang, Jun-Hao Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

TaRA: Training-Aware Low-Rank Adaptation Initialization

Training-aware Low-Rank Adaptation Initialization (TaRA) is proposed, a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix.

Taehyeon Kim, Eunhyeok Park · 0 citations
#machine learning Preprint Sep 2026

LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry

Extending LILA to dynamically allocate sparsity budgets via KS-scores yields state-of-the-art generative preservation at moderate compression, while uncovering fundamental single-layer architectural bottlenecks at higher compression regimes.

Sankar Behera, Dhruv Singh, Anshika Agnihotri et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.