#machine learning
Apr 2026
Attention to Mamba: A Recipe for Cross-Architecture Distillation
This work proposes a principled two-stage approach to distill knowledge from a traditional Transformer into a linearized version of Attention, using an adaptation of the kernel trick, which retains most of the original Pythia-1B Transformer performance.
A. Moudgil, N. Huang, Eeshan Gunesh Dhekane et al.
· arXiv.org · 3 citations
· ⚡1