Aug 2026· International Conference on Computer Vision and Information Technology· Vol 14321, pp. 143210B - 143210B-5· 0 citations· 9 references
Engineering
TL;DR
This work evaluates Discrepancy-Guided Parameter Suppression on the challenging iWildCam dataset under camera-induced domain shifts, providing the first in-depth assessment of DPS on a real-world distribution shift benchmark.
Abstract
Fine-tuning large pre-trained models on specific tasks can improve in-distribution (ID) accuracy but often degrades out-of-distribution (OOD) generalization. Discrepancy-Guided Parameter Suppression (DPS) is a fine-tuning approach that Discrepancy-Guided Parameter Suppression (DPS) is a fine-tuning approach that freezes the top K percent of model parameters based on the magnitude of their changes. By ranking parameters according to a discrepancy score (the magnitude of change during fine-tuning) and suppressing the highest-discrepancy parameters, DPS encourages the model to learn task-relevant features without overfitting to spurious ID-specific patterns. We evaluate DPS on the challenging iWildCam dataset under camera-induced domain shifts, providing the first in-depth assessment of DPS on a real-world distribution shift benchmark. Using a CLIP ViT-B/16 backbone and identical hyperparameters for a cross-entropy baseline and DPS, we observe improvements on OOD metrics: OOD macro-F1 increases from 35.14% to 36.15% and OOD accuracy from 73.92% to 74.95%, roughly a 1% absolute gain on both. We also provide ablation analyses of the discrepancy metric and suppression rate, showing that normalized, channel-wise suppression at high freezing ratios yields the strongest OOD robustness, consistent with prior findings.
Class-wise Covariance Regularization is proposed, which aligns the predicted covariance structure of class confidences with the semantic correlations encoded in pretrained text embed-dings with the geometric consistency of the class space throughout fine-tuning, resulting in more stable and interpretable confidence distributions across categories.
Ao Zhou, Zhiwei Jiang, Zifeng Cheng et al.· 0 citations
Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support-level design does not explicitly select sparse, structured regions that are task-salient for downstream adaptation. To address this limitation, we propose SPIRA, a PEFT framework that separates support discovery from parameter-efficient adaptation. During a short warm-up, the Relative Information-Geometric Potential (RIGP) identifies a high-saliency sparse seed support from a squared-gradient base statistic. This online criterion draws on the local-sensitivity perspective of information geometry without constructing the full Fisher information matrix. The selected seed support determines active input and output indices, which in turn define a structural closure. SPIRA fixes this closure as the adaptation mask and trains active-axis low-rank factors whose parameter count scales with the active dimensions. Experiments across computer vision, natural language processing, and vision-language modeling benchmarks show that SPIRA remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhongyi Wen, Zhikai Zhai, Guomin Sun et al.· IEEE Transactions on Pattern...· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Kwame Nkosi· International Journal of App...· 0 citations
In this paper, we propose AdaMSS, an adaptive multi-subspace approach for parameter-efficient fine-tuning of large models. Unlike traditional parameter-efficient fine-tuning methods that operate within a large single subspace of the network weights, AdaMSS leverages subspace segmentation to obtain multiple smaller subspaces and adaptively reduces the number of trainable parameters during training, ultimately updating only those associated with a small subset of subspaces most relevant to the target downstream task. By using the lowest-rank representation, AdaMSS achieves more compact expressiveness and finer tuning of the model parameters. Theoretical analyses demonstrate that AdaMSS has better generalization guarantee than LoRA, PiSSA, and other single-subspace low-rank-based methods. Extensive experiments across image classification, natural language understanding, and natural language generation tasks show that AdaMSS achieves comparable performance to full fine-tuning and outperforms other parameter-efficient fine-tuning methods in most cases, all while requiring fewer trainable parameters. Notably, on the ViT-Large model, AdaMSS achieves 4.7% higher average accuracy than LoRA across seven tasks, using just 15.4% of the trainable parameters. On RoBERTa-Large , AdaMSS outperforms PiSSA by 7% in average accuracy across six tasks while reducing the number of trainable parameters by approximately 94.4%. These results demonstrate the effectiveness of AdaMSS in parameter-efficient fine-tuning. The code for AdaMSS is available at https: //github.com/jzheng20/AdaMSS .
Jingjing Zheng, Wanglong Lu, Yiming Dong et al.· Neural Information Processin...· 0 citations
Standard supervised fine-tuning (SFT) assigns the same explicit loss weight to every expert demonstration, regardless of the model's changing competence over training queries. Reinforcement learning (RL) based methods adapt update strength using model-generated rollouts, but often require substantially more sampling and can be unstable on hard tasks. We propose \textbf{Online Self-Weighted Fine-Tuning (OSW-FT)}, a simple method that augments SFT with online, trajectory-level weighting. For each query, OSW-FT estimates the model's current success rate using a small number of inference-only rollouts and rescales the standard SFT loss accordingly. The optimization direction remains anchored to the expert trajectory, while the update magnitude adapts online. For binary-verifiable reasoning, we connect this weighting to SFT and RL at the gradient level, inspired by variance-reduction principles. The resulting estimator is unbiased for the exact OSW-FT surrogate update for any finite rollout count, and we analyze convergence with respect to the corresponding surrogate objective. Evaluated across Qwen3 series ranging from 0.6B to 4B on multiple challenging benchmarks (e.g., AIME), OSW-FT consistently improves over SFT on small-to-medium scale models. OSW-FT offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only \textbf{2 online rollouts}.
Hai-Quan Wen, Yiwei He, Bei Peng et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
