Jul 2026
On the Convergence of Stochastic Low-Rank Adaptation
This work proposes LoRA-NSGDM, which finds an $\epsilon$-stationary point with $\mathcal{O}(\epsilon^{-8})$ stochastic oracle complexity, and LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}(\epsilon^{-6})$.
Ru Wang, Chengchang Liu, John C. S. Lui
· arXiv.org · 1 citation