SPIRA: Sparse Information-Geometric Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Pretrained Models.
TL;DR
SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Abstract
Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support-level design does not explicitly select sparse, structured regions that are task-salient for downstream adaptation. To address this limitation, we propose SPIRA, a PEFT framework that separates support discovery from parameter-efficient adaptation. During a short warm-up, the Relative Information-Geometric Potential (RIGP) identifies a high-saliency sparse seed support from a squared-gradient base statistic. This online criterion draws on the local-sensitivity perspective of information geometry without constructing the full Fisher information matrix. The selected seed support determines active input and output indices, which in turn define a structural closure. SPIRA fixes this closure as the adaptation mask and trains active-axis low-rank factors whose parameter count scales with the active dimensions. Experiments across computer vision, natural language processing, and vision-language modeling benchmarks show that SPIRA remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.