Continual Low-Rank Adaptation Via Cumulative Unified Optimization.

Yue Lu Shizhou Zhang De Cheng Y. Xing Guoqiang Liang Nannan Wang Peng Wang Yanning Zhang

Aug 2026 · IEEE Transactions on Pattern Analysis and Machine Intelligence · Vol PP, pp. 1-18 · 0 citations

Medicine

TL;DR

This work reformulates LoRA-based CL as a consistent feature mapping problem that mimics the behavior of the joint-training upper bound, wherein a unified adaptation parameter matrix is learned to simultaneously capture the input-output relationships established by all task-specific LoRAs.

Abstract

Low-Rank Adaptation (LoRA) enables foundation models to efficiently adapt to downstream tasks with minimal trainable parameters, yet it lacks the capability of continual learning (CL) from non-stationary data streams. While existing CL methods attempt to address this by storing and retrieving task-specific parameter modules, they suffer from non-trivial challenges of task identification and inconsistency in input-output feature mappings, resulting in suboptimal feature representations. Instead of keeping multiple task-specific parameter modules, we propose a novel continual LoRA mechanism that parameterizes all task-specific adaptations with a single adaptation module, naturally obviating the task identification and mapping inconsistency issues. To this end, we reformulate LoRA-based CL as a consistent feature mapping problem that mimics the behavior of the joint-training upper bound, wherein a unified adaptation parameter matrix is learned to simultaneously capture the input-output relationships established by all task-specific LoRAs. By solving this formulation, our approach facilitates approximate unified optimization over sequential tasks via progressively updating cumulative feature statistics, achieving efficient knowledge retention and renewal. Notably, the proposed arithmetic training mechanism can be applied both to the backbone of the foundation model for parameter-efficient fine-tuning and the classifier optimization, thereby enabling continual learning within an integrated architecture that closely approximates the joint learning paradigm. Extensive experiments across conventional and long-sequence class-incremental benchmarks demonstrate that our method achieves new state-of-the-art performance.

DOI PubMed Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

TaRA: Training-Aware Low-Rank Adaptation Initialization

Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.

Taehyeon Kim, Eunhyeok Park · 0 citations

SeMi-LoRA: Enhancing Low-Rank Adaptation via Separation and Mixing

Zhenfei Yang, Beiming Yu, Peiqin Lin et al. · 0 citations

#small language model Preprint Aug 2026

Unifying Detection and Adaptation in Task-Free Continual Learning

Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.

Dezheng Han, Anlan Zhang, Zhiwu Zhu et al. · 0 citations

Preprint Aug 2026

Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and repetitive collapse of task-specific updates into previously occupied directions. We introduce Geo-LoRA, a geometry-aware framework that explicitly regulates how low-rank subspaces, both shared and task-specific, evolve during continual learning. For the shared branch, Subspace Projection Preservation (SPP) constrains consecutive updates to follow smooth trajectories on the Grassmann manifold, and Adaptive Core-Slack Alignment (ACSA) decomposes transitions into principal and residual components, aligning the former while modulating the latter to balance stability and plasticity. For the task-specific branch, Median-Calibrated Block Overlap (MCBO) imposes a statistical constraint via normalized projection overlap, penalizing excessive reuse to mitigate subspace crowding. These constraints jointly regulate the evolution of all LoRA subspaces across layers and tasks without introducing additional adapter types beyond standard LoRA. Geo-LoRA provides a principled geometric formulation for continual low-rank adaptation and consistently achieves state-of-the-art performance across multiple benchmark datasets and different task lengths.

Yihui Feng · 0 citations

Preprint Aug 2026

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space, which fosters robust, task-agnostic features without explicit partitioning overhead.

Qian Zha, Jinda Liu, Yuan Wu et al. · 1 citation

Preprint Aug 2026

Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning

This work introduces a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks, and allows the update of parameters with high past-task importance when they have low relative-importance, thus enabling backward knowledge transfer.

Malavika Suresh, I. Nkisi-Orji, N. Wiratunga · 0 citations