Parameter-Efficient Fine-Tuning of Transformer Models: An Empirical Study of LoRA Optimization for High-Cardinality Intent Classification

Abstract

Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed to remedy such performance degradation by defining the state-of-the-art low-rank adaptation. Instead of being constrained by the conventional formulation, this approach uses the minimal possible rank-8 adapter, full linear module coverage, and a learned learning rate. Extensive ablations offer two important discoveries that structural module coverage has more impact than mere adapter rank, and high learning rate is indispensable to provide enough convergence with the limited number of parameters. We show in the experiment that this configuration manages to restore the model performance to the 93.73% and 90.18% on Banking77 and CLINC150 respectively. S1 configuration attains the baseline parity, while only updating 1.56%-1.60% total trainable parameters with a maximal 2.96GB memory. This proves that it is possible to train a high-accuracy transformer on the hardware targeted at consumers, for example the NVIDIA RTX 5060 Ti.