Code-Switch Pretraining for Improved Cross-Lingual Alignment in Low-Resource Languages
This work proposes an approach that integrates code-switching directly into masked language model pretraining, and introduces a multiview probabilistic translation strategy that samples candidate translations based on alignment likelihoods, applying substitutions only to unmasked tokens.
Ruan Visser, Trienko L. Grobler, Marcel Dunaiski
· 0 citations