MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum
MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed) that ranks first among student models on the MTEB(Turkish) overall average and outperforms all compared Turkish tokenizers on compression and fertility across two independent test sets.
Furkan Yılmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
· 0 citations