Skip to content

IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning

Jul 2026 · arXiv.org · Vol abs/2607.22251 · 0 citations · 46 references
Computer Science

TL;DR

IFCLoRA is proposed, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization and achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage.

Abstract

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT) of LLMs whose effectiveness depends on rank allocation. Existing adaptive LoRA methods derive ranks from local gradient, activation, or matrix statistics collected before or during fine-tuning; training-time variants add overhead, and local signals reveal little about each module's structural role in information propagation, giving weak global grounding for scarce-capacity allocation. We propose IFCLoRA, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization. Using a small calibration set, IFCLoRA performs intervention tracing on the frozen model and constructs a sparse task-conditioned interaction graph over LoRA target modules. From this graph it extracts a global information-flow topology prior and fuses it with each node's local gradient sensitivity to form a topology-dominant Information-Flow Centrality (IFC) score, measuring participation in task-conditioned multi-hop propagation. The IFC scores then serve as module-level routing signals for one-shot discrete rank allocation under a rank-budget constraint. Reusing response vectors from tracing, IFCLoRA constructs a function-preserving flow-response subspace initialization, giving adapters task-relevant output subspaces. Across all settings, IFCLoRA achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage. On GSM8K, IFCLoRA attains the highest mean accuracy among compared PEFT methods on both base models, exceeding standard LoRA by 4.75 percentage points on LLaMA-3.1-8B. Resulting rank allocations are non-uniform and vary across tasks and base models, suggesting that task-conditioned global information-flow topology can serve as a useful structural prior for rank allocation in low-budget PEFT.

View source

Similar papers

Open access Aug 2026

Contextual Bandit Routing for LoRA Experts

The deployment of multiple Low-Rank Adaptation (LoRA) experts for domain-specific tasks requires an efficient routing mechanism. Existing approaches rely on offline-trained routers or static heuristics, which fail to adapt to unseen domains or shifting query distributions. We propose EXP4-LoRA, a contextual bandit fram...

Miavaka Voninahitra Ambinintsoa Rakotovao Valisoa, Hobihery Matio Robinson · 0 citations
Aug 2026

SPIRA: Sparse Information-Geometric Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Pretrained Models.

SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.

Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al. · 0 citations
Preprint Aug 2026

CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning

By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space, which fosters robust, task-agnostic features without explicit partitioning overhead.

Qian Zha, Jinda Liu, Yuan Wu et al. · 2 citations
#artificial intelligence Preprint Sep 2026

TaRA: Training-Aware Low-Rank Adaptation Initialization

Training-aware Low-Rank Adaptation Initialization (TaRA) is proposed, a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix.

Taehyeon Kim, Eunhyeok Park · 0 citations
Open access Sep 2026

TAST: Task-Aware Sparse Topology Learning for Multi-Task Recommendation

TAST is proposed, a Task-Aware Sparse Topology Learning framework that constructs task-specific subnetworks within a shared network that learns shared parameters and task-specific topologies, hardens the learned masks into binary structures, and optionally applies a lightweight sample-conditioned refinement module for...

Ming-Zhu Zhang, Zhen Yang, Rui-Ping Yin · 0 citations
#artificial intelligence Preprint Aug 2026

RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks

A random-walk-based LoRA fine-tuning scheme that eliminates the need for global synchronization, substantially reduces communication and computation costs, and avoids aggregation errors is proposed and provides rigorous convergence guarantees for non-convex objectives under standard assumptions.

Xing-Ran Chen, Rohit Bhagat, Ghadir Ayache et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.