Preprint

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

Qian Zha Jinda Liu Yuan Wu Yi Chang

Aug 2026 · 1 citation · 35 references

Computer Science

TL;DR

By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space, which fosters robust, task-agnostic features without explicit partitioning overhead.

Abstract

While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.

arXiv Semantic Scholar

View source

Similar papers

Representation-Aware Modularity: Efﬁcient Cross-Task Generalization for LLMs

RaMod, a Representation-Aware Modularity framework, is proposed, a Representation-Aware Modularity framework to extend the ReFT paradigm to CTG through two novel components: Dual-Modular Representation & Parameter Fine-tuning, which manipulates only a strategically chosen subset of hidden representations with modular interventions to guide the model toward solving unseen tasks.

Zheng Gong, Ying Sun, Chao Wang et al. · 0 citations

Jul 2026

The Parts Are Greater Than the Sum: Automated Task Sequencing for Efficient Training of Multi-Policy LLMs

Performance consistently improves from conventional single-policy PEFT to multi-policy PEFT, with the proposed automatic multi-policy framework achieving the best performance under the same trainable capacity, suggesting that optimization-path organization is more effective than simply increasing adapter capacity for heterogeneous parameter-efficient fine-tuning.

Jia-Jia Tang, Si-Zhe Yuen, Francisco Gomez Medina et al. · arXiv.org · 0 citations

SeMi-LoRA: Enhancing Low-Rank Adaptation via Separation and Mixing

Zhenfei Yang, Beiming Yu, Peiqin Lin et al. · 0 citations

Aug 2026

GLA-LoRA: Parameter-efficient LLM fine-tuning with global-local knowledge alignment.

GLA-LoRA establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation and establishes that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.

Hao Wu, Jianqi Gao, Xiangfeng Luo · Neural Networks · 0 citations

Jul 2026

MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation

This paper makes the first attempt to fine-tune MoE models with MoE-style low-rank adaptation via a dual-channel Routing-Conditioned Projection module, which reuses base router activations to inform LoRA routing and introduces a single global LoRA expert pool shared across all layers.

Qingyu Yang, Haonan He, Minglei Li et al. · arXiv.org · 0 citations

Aug 2026

Continual Low-Rank Adaptation Via Cumulative Unified Optimization.

This work reformulates LoRA-based CL as a consistent feature mapping problem that mimics the behavior of the joint-training upper bound, wherein a unified adaptation parameter matrix is learned to simultaneously capture the input-output relationships established by all task-specific LoRAs.

Yue Lu, Shizhou Zhang, De Cheng et al. · IEEE Transactions on Pattern... · 0 citations