2026· International Conference on Conceptual Structures· pp. 294-308· 0 citations· 39 references
Computer Science
TL;DR
These findings suggest that direct configuration prediction provides a robust approach despite ambiguity in the parameter space, and are compared to a regression-based performance prediction model, a multiple-choice model that treats near-optimal configurations as valid targets, and a baseline single-label model.
Parameter-efficient fine-tuning (PEFT) methods have become essential for adapting large language models (LLMs) to downstream tasks without updating all parameters. However, the optimal PEFT strategy varies significantly across model architectures, task types, and resource constraints, making method selection a non-trivial problem. This paper presents PEFT-Assess, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability. We benchmark these methods on LLaMA-7B and Qwen-7B across six NLP tasks from the GLUE and SuperGLUE benchmarks. Furthermore, we propose a gradient boosting-based strategy selector that predicts the optimal PEFT method given task characteristics and resource budgets. Experiments show that LoRA achieves the best overall adaptability score of 0.847, while our strategy selector attains 91.2% prediction accuracy for recommending the optimal PEFT method.
Zhaoyang Hu, Shixuan Cao, Yuqi Sun et al.· International Conference on...· 0 citations
Evolutionary Soups, a mixture-of-experts framework for fine-grained generation control, with gating networks trained via an evolutionary algorithm, achieves the best hypervolume, linear utility, and Tchebyshev utility among controllable methods on all tasks.
Lingxiao Kong, Steffen Staab, Cong Yang et al.· 0 citations
The In-context learning (ICL) paradigm aids large language models (LLMs) to adapt to new tasks without need for fine-tuning. However, selecting an optimal combination of demonstration examples from a large pool of example subsets is a challenging problem. Existing approaches for selection do not model the complex relationship between ICL samples and downstream LLM performance. They typically perform static task-level selection, choosing subsets once offline, which can fail to generalize to unseen queries. We introduce DearICL (Data Efficient Algorithm for Ranking) ICL samples, a new framework that models demonstration example selection as a subset ranking problem. DearICL employs a non-linear surrogate employing a differentiable sorting objective within a gap-index bandit algorithm. The gap-index based approach enables fine-grained separation of good arms and borderline arms, which is used as an auxiliary objective to train the non-linear surrogate through sufficient sampling of borderline arms, supporting instance-level subset ranking. On exemplar selection benchmarks with open-source LLMs, DearICL achieves 8.08-15.9% accuracy gains over strong linear bandit baselines, with low sample complexity. Code and data: https://github.com/VenkteshV/DearICL.
Experiments show that DIAG boosts yield across iterations and delivers stronger reasoning performance under an iso-effective training budget, demonstrating that it can distill more informative preference supervision for mathematical reasoning.
Guhan Chen, Songtao Tian, Bo-Han Li et al.· 0 citations
Model Internal State Optimization (MISO), a systems workflow that uses model internal states (MIS), including parameters, activations, gradients, and normalization statistics, to prioritize such local optimization decisions.
Yongzhen Zhang, Xiaoyu Deng, Yifan He et al.· 0 citations
Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed to remedy such performance degradation by defining the state-of-the-art low-rank adaptation. Instead of being constrained by the conventional formulation, this approach uses the minimal possible rank-8 adapter, full linear module coverage, and a learned learning rate. Extensive ablations offer two important discoveries that structural module coverage has more impact than mere adapter rank, and high learning rate is indispensable to provide enough convergence with the limited number of parameters. We show in the experiment that this configuration manages to restore the model performance to the 93.73% and 90.18% on Banking77 and CLINC150 respectively. S1 configuration attains the baseline parity, while only updating 1.56%-1.60% total trainable parameters with a maximal 2.96GB memory. This proves that it is possible to train a high-accuracy transformer on the hardware targeted at consumers, for example the NVIDIA RTX 5060 Ti.
Harsh Anand, Sofia Singh, Rahul Agrawal et al.· 2026 International Conferenc...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
