IFCLoRA is proposed, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization and achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage.
Abstract
Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT) of LLMs whose effectiveness depends on rank allocation. Existing adaptive LoRA methods derive ranks from local gradient, activation, or matrix statistics collected before or during fine-tuning; training-time variants add overhead, and local signals reveal little about each module's structural role in information propagation, giving weak global grounding for scarce-capacity allocation. We propose IFCLoRA, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization. Using a small calibration set, IFCLoRA performs intervention tracing on the frozen model and constructs a sparse task-conditioned interaction graph over LoRA target modules. From this graph it extracts a global information-flow topology prior and fuses it with each node's local gradient sensitivity to form a topology-dominant Information-Flow Centrality (IFC) score, measuring participation in task-conditioned multi-hop propagation. The IFC scores then serve as module-level routing signals for one-shot discrete rank allocation under a rank-budget constraint. Reusing response vectors from tracing, IFCLoRA constructs a function-preserving flow-response subspace initialization, giving adapters task-relevant output subspaces. Across all settings, IFCLoRA achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage. On GSM8K, IFCLoRA attains the highest mean accuracy among compared PEFT methods on both base models, exceeding standard LoRA by 4.75 percentage points on LLaMA-3.1-8B. Resulting rank allocations are non-uniform and vary across tasks and base models, suggesting that task-conditioned global information-flow topology can serve as a useful structural prior for rank allocation in low-budget PEFT.
The deployment of multiple Low-Rank Adaptation (LoRA) experts for domain-specific tasks requires an efficient routing mechanism. Existing approaches rely on offline-trained routers or static heuristics, which fail to adapt to unseen domains or shifting query distributions. We propose EXP4-LoRA, a contextual bandit fram...
SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al.· IEEE Transactions on Pattern...· 0 citations
By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space, which fosters robust, task-agnostic features without explicit partitioning overhead.
Training-aware Low-Rank Adaptation Initialization (TaRA) is proposed, a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix.
TAST is proposed, a Task-Aware Sparse Topology Learning framework that constructs task-specific subnetworks within a shared network that learns shared parameters and task-specific topologies, hardens the learned masks into binary structures, and optionally applies a lightweight sample-conditioned refinement module for...
A random-walk-based LoRA fine-tuning scheme that eliminates the need for global synchronization, substantially reduces communication and computation costs, and avoids aggregation errors is proposed and provides rigorous convergence guarantees for non-convex objectives under standard assumptions.
Xing-Ran Chen, Rohit Bhagat, Ghadir Ayache et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.