Skip to content

Author

P. Rodríguez

1 paper indexed here

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

Attention to Mamba: A Recipe for Cross-Architecture Distillation

This work proposes a principled two-stage approach to distill knowledge from a traditional Transformer into a linearized version of Attention, using an adaptation of the kernel trick, which retains most of the original Pythia-1B Transformer performance.

A. Moudgil, N. Huang, Eeshan Gunesh Dhekane et al. · 3 citations · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.