Overcoming Data Imbalances with Expert Selection Biases
Mixture-of-Experts (MoE) models have become the standard architecture for frontier LLMs. The multiple parallel experts in each transformer block theoretically enable expert specialization according to categories such as domains, languages, or modalities during pretraining. However, when data is imbalanced across catego...
A. Tarun, Clara Meister, Antoine Bosselut
· 0 citations