Attention to Mamba: A Recipe for Cross-Architecture Distillation
This work proposes a principled two-stage approach to distill knowledge from a traditional Transformer into a linearized version of Attention, using an adaptation of the kernel trick, which retains most of the original Pythia-1B Transformer performance.