Using Weighted Loss in Bilingual Model Training

Abstract

Large Language Models (LLMs) have achieved remarkable performance largely due to the availability of high-quality and large-scale training data. However, this reliance creates a significant performance disparity between high-resource languages such as English and low-resource languages like Arabic. The imbalance in data exposure during the pre-training phase persists into the bilingual supervised fine-tuning (SFT) stage, limiting the models’ semantic depth and reasoning capabilities in low-resource languages.To address the aforementioned imbalance, we depart from traditional approaches by computing the optimal learning rates for each language in the dataset separately and transforming these learning rates into language-specific weighting coefficients used in the loss function. Experimental results obtained on the XQuAD test set after training on the MURI dataset demonstrate that the proposed method preserves performance in English while significantly improving Arabic performance, achieving 56.36 F1 and 40.27 Exact Match (EM) scores.