Research on Efficient Fine-Tuning of Large Model Parameters Using a Hybrid LoRA and IA3 Adaptation Approach

Abstract

Addressing the high computational cost, large GPU memory consumption, and limited cross-task generalization of full-parameter fine-tuning for Large Language Models (LLMs), especially when adapting them to specialized engineering domains such as electromagnetic wave analysis, antenna design, and propagation scenario modelling, this paper proposes a parameter-efficient fine-tuning method named LoRA-IA3. The method combines Low-Rank Adaptation (LoRA) with Infused Adapter by Inhibiting and Amplifying Inner Activations (IA3), and achieves task adaptation while preserving the general knowledge of the pre-trained model through a dual mechanism of low-rank matrix decomposition and activation inhibition-amplification. First, the LoRA module is introduced into the attention layer of the Transformer architecture to perform low-rank decomposition on key weight matrices, thereby reducing the number of trainable parameters. Second, the IA3 adapter is embedded in the Feed-Forward Network (FFN) layer to dynamically adjust activation distributions through per-channel scaling factors, enhancing task-specific feature extraction. Finally, to maintain stable training and fast convergence when adapting models to complex electromagnetic engineering tasks, including antenna parameter interpretation, propagation data analysis, and technical text understanding, a hybrid adaptation training strategy is designed using hierarchical learning rates and gradient clipping mechanisms across different modules. The proposed approach provides an efficient fine-tuning framework for applying LLMs to domain-specific engineering tasks with reduced computational overhead.