Wen-Qing Su

Efficient sparse low-rank adaptation for large language models.

This paper investigates SoRA algorithm's performance in large-scale models, and derives an improved learning rate configuration and proposes SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally and mitigates the suboptimality of the original SoRA method.