Efficient Pre-Training of LLMs through Truncated SVD Representations
The TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching is presented, and a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices.
Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad et al.
· 0 citations