Latent Core Tokenizer: Compress, but Meaningfully
Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Leng...
Felermino D. M. A. Ali, Millicent Ochieng, O. Ekwejunor-Etchie et al.
· 0 citations