Lara Mini Base: Language Diversity as a Resource, Not a Trade-off
Lara Mini base is presented, a model trained from scratch on 6T tokens in English and 44 other languages, which together cover all 24 official languages of the European Union, and it is found that the MoE architecture reduces capacity competition between languages.
Translated, M. Adinolfi, Isabelle Andrieu et al.
· 0 citations