The results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
Abstract
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
This work provides a new method for fine-tuning models with sparse attention that works for any KV cache policy, runs on a moderate hardware budget, and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism).
Matthias W. Seeger, Zeyu Zhang, Vihang Patil et al.· 0 citations
Parameter-efficient fine-tuning (PEFT) methods like LoRA have become crucial for adapting large language models under computational constraints. While LoRA focuses on dense weight updates, recent sparse methods propose updating only selected portions of pretrained weights. However, these sparse approaches predominantly use element-wise parameter selection, overlooking inherent structured patterns in LLM weight distributions. We introduce StructTune1, a PEFT method that uses entire rows or columns as basic units for identifying task-relevant sparse components. Our analysis reveals LLaMA models exhibit distinct structured patterns in pretrained weights, with fine-tuning updates showing structured amplification rather than uniform changes. We develop a sensitivity-based framework to efficiently identify task-relevant weight structures without full fine-tuning. Experiments on mathematical reasoning and code generation demonstrate StructTune’s structured sparse updates outperform both dense LoRA and element-wise sparse methods while maintaining computational efficiency. Ablation studies validate our sensitivity-based selection strategy, confirming the efficacy of exploiting structured weight patterns in PEFT.
Unknown authors· Transactions of the Associat...· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
F-WANDA is introduced, a drop-in modification of WANDA that reallocates the per-row keep budget across output neurons in proportion to the empirical Fisher information of the pre-activation, placing F-WANDA on the Pareto frontier of quality versus pruning cost for sustainable LLM compression.
RaMod, a Representation-Aware Modularity framework, is proposed, a Representation-Aware Modularity framework to extend the ReFT paradigm to CTG through two novel components: Dual-Modular Representation & Parameter Fine-tuning, which manipulates only a strategically chosen subset of hidden representations with modular interventions to guide the model toward solving unseen tasks.
Zheng Gong, Ying Sun, Chao Wang et al.· 0 citations
Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.
Shi-Cheng Wei· Theoretical and Natural Scie...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
