Circuit Fine-Tuning is introduced, a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training to isolate the response of the backbone to the target distribution rather than the preferences of a particular classifier.
Abstract
Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant efficiency metric in PEFT, it does not imply \textit{compute efficiency}: parameter-sparse methods can still incur full-model training cost per step, and typically need long schedules to reach peak accuracy. We introduce Circuit Fine-Tuning (CFT), a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training. Whereas attribution is conventionally formulated against a trained task head, we formulate it against a near-zero-initialized probe head, which isolates the response of the backbone to the target distribution rather than the preferences of a particular classifier. CFT then fine-tunes only the recovered subgraph. CFT needs no learning-rate warmup and reaches peak accuracy in ${\sim}20$ epochs on average---versus $44$--$96$ for strong PEFT baselines---yielding $2.3$--$6.6\times$ fewer training FLOPs and up to $16\times$ less wall-clock time, while adding zero parameters and no inference operations. Experiments across a standard visual transfer benchmark (VTAB-1k), hierarchical backbones (Swin), domain-shifted medical imaging (CBIS-DDSM), and a vision-language model (Gemma-3 on CUB-200) demonstrate the effectiveness of CFT. Code is available at https://github.com/UriKialy/CFT
The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Harshavardhan Adepu, Li Zhang, Sanjiv Kumar et al.· 0 citations
Vision Transformer (ViT) obtains state-of-the-art performance but is expensive to fine-tune because of the large number of parameters. Parameter-efficient tuning methods, such as LoRA, have been proposed. However, existing methods tend to ignore the hierarchical structure of representation learning and assume the same distributed-rank assignment for all layers. This paper offers a rank assignment technique for LoRA finetuning of Vision Transformer with layer-wise differential ranking rather than uniform, which gives better channel utilization efficiency. The test used the CIFAR-100 dataset with the ViT Tiny model. The testing results show that the method can achieve 0.8086 average accuracy while decreasing the tunable parameter ratio to 2.43% from the original 4.09%, which is an advantage over LoRA with its uniform rank. Additionally, the method is seed-agnostic, and thus, the approach is reasonably stable. These findings suggest that placing different hierarchies increases parameter efficiency; hence, it is a viable method that can be used to efficiently fine-tune Vision Transformers.
The results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.
Ivan D. Ilin, Philip Zmushko, Peter Richtárik· arXiv.org· 0 citations
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.
Valentijn Oldenburg, F. D. Kam, Bente Zuijdam et al.· 0 citations
Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed to remedy such performance degradation by defining the state-of-the-art low-rank adaptation. Instead of being constrained by the conventional formulation, this approach uses the minimal possible rank-8 adapter, full linear module coverage, and a learned learning rate. Extensive ablations offer two important discoveries that structural module coverage has more impact than mere adapter rank, and high learning rate is indispensable to provide enough convergence with the limited number of parameters. We show in the experiment that this configuration manages to restore the model performance to the 93.73% and 90.18% on Banking77 and CLINC150 respectively. S1 configuration attains the baseline parity, while only updating 1.56%-1.60% total trainable parameters with a maximal 2.96GB memory. This proves that it is possible to train a high-accuracy transformer on the hardware targeted at consumers, for example the NVIDIA RTX 5060 Ti.
Harsh Anand, Sofia Singh, Rahul Agrawal et al.· 2026 International Conferenc...· 0 citations
Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support-level design does not explicitly select sparse, structured regions that are task-salient for downstream adaptation. To address this limitation, we propose SPIRA, a PEFT framework that separates support discovery from parameter-efficient adaptation. During a short warm-up, the Relative Information-Geometric Potential (RIGP) identifies a high-saliency sparse seed support from a squared-gradient base statistic. This online criterion draws on the local-sensitivity perspective of information geometry without constructing the full Fisher information matrix. The selected seed support determines active input and output indices, which in turn define a structural closure. SPIRA fixes this closure as the adaptation mask and trains active-axis low-rank factors whose parameter count scales with the active dimensions. Experiments across computer vision, natural language processing, and vision-language modeling benchmarks show that SPIRA remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhongyi Wen, Zhikai Zhai, Guomin Sun et al.· IEEE Transactions on Pattern...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
