MoRE: Mixture of Reused Experts
Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling...
Eric S. Qiu, U. U. Acikalin, Justin Lovelace et al.
· 1 citation