Stabilizing the Dynamic Low-Rank Training
The gradient flow of the best rank-$r$ approximation is derived and it is pointed out that the offset of DLRT comes from a curvature-coupling term which is large and thus non-negligible under aggressive compression, and a stable dynamic low-rank training method is proposed, named SDLRT.
Zhong-Han Xu, Ling Wang, Jun-Hao Chen et al.
· 0 citations