Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
A linguistically motivated tokenizer for Vietnamese and Chinese that converts each syllable into IPA and factorizes it into three phonological components: onset, rime, and tone is introduced.
Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le et al.
· 0 citations