Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG

Abstract

Addressing the resource constraints of a single NVIDIA RTX 5000 (16 GB) GPU, this applied study takes DeepSeek-LLM-7B-Base as the base model and systematically compares four parameter-efficient fine-tuning methods: LoRA, QLoRA, DoRA, and LoRA+. It also validates a Retrieval-Augmented Generation (RAG) architecture tailored for zero-tolerance engineering specifications. Experiments are conducted on a private construction process dataset. Theoretical analysis shows that the low-rank assumption of LoRA originates from the intrinsic dimensionality property of pre-trained models; LoRA+ adopts an asymmetric learning rate strategy (with the optimal ratio ηAηB = 0.05 determined via grid search), effectively solving the suboptimal training dynamics problem of standard LoRA caused by a uniform learning rate; DoRA decomposes weight updates into magnitude and direction components on a spherical manifold and a positive real manifold; RAG guarantees hallucination suppression through the conditional entropy inequality H(Y|Q,D,θ) ≤ H(Y|Q,θ). Experimental results demonstrate that LoRA+ outperforms other baseline methods in BLEU-4 (0.5609), ROUGE-L (0.5387), and PPL (2.1433), with a training time of 1.8 h and memory usage of 13.1 GB. After introducing RAG on top of LoRA+, BLEU-4 further improves to 0.5814, ROUGE-L to 0.5557, and the hallucination rate(HR) drops from 1.71% to 0.08%, achieving an Exact Match (EM) score of 0.2778 and high traceability (Recall@3 = 0.9961). This study provides a technical pathway and empirical evidence for deploying large models in the construction domain under resource-constrained conditions through the synergy of fine-tuning and RAG.