GME-Init: Gamma-Moment Equalization for LoRA Initialization in Parameter-Efficient Fine-Tuning

Abstract

Low-Rank Adaptation (LoRA) is a representative parameter-efficient fine-tuning method that reduces computational and memory costs without modifying the model architecture. Standard LoRA initializes matrix A from a symmetric distribution, such as Gaussian or Kaiming initialization, and matrix B to zero. Although this provides a statistically neutral starting point, it ignores the influence of task-specific input features on initialization. We propose Gamma-Moment Equalization Initialization (GME-Init), a data-aware asymmetric LoRA initialization method based on output-moment calibration. Using a small calibration set, GME-Init estimates the variance and skewness of target-layer outputs and adjusts the layer-wise initialization scale and asymmetry of LoRA weights, improving their statistical alignment with task-specific skewed representations. GME-Init operates only during initialization and does not change the LoRA architecture, trainable parameter count, training budget, or inference cost. We evaluate it on a GLUE subset with RoBERTa-base, integrate it with AdaLoRA and DoRA, and test it on VRSBench-VQA, VRSBench-Caption, and UCM-Caption using Qwen2.5-VL-3B-Instruct. Results show that GME-Init serves as a simple plug-in PEFT initialization module with no additional inference cost and that it consistently improves standard LoRA and selected LoRA-style methods across the evaluated text understanding and multimodal tasks.