Sequential Functional Structured Tucker Compression for Large Language Model Attentions
FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget, achieves the lowest WikiText-2 perplexity among the compared methods.
Jian Chen, Xin-Yu Wang, Tian-Shuo Yan et al.
· 0 citations