Ze-Bang Xie

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to ma...