It is demonstrated that multilingual embedding models provide a more effective and scalable solution for cross-lingual retrieval-augmented generation (RAG) in low-resource government domains.
Abstract
This paper presents a comparative evaluation of cross-lingual information retrieval (CLIR) methods for retrieving English government information using Sinhala and Tamil queries. Two CLIR paradigms are investigated: Query Translation (QT), employing Google Translate, NLLB, and mBART50, and Cross-Lingual Embeddings (CLE), using LaBSE, multilingual E5, and BGE-M3, with monolingual English retrieval as the baseline. Experiments are conducted on a human-verified benchmark comprising 500 Sinhala, Tamil, and English question-answer pairs derived from 1,699 segmented contexts from Sri Lanka's Government Information Center (GIC). Retrieval performance is evaluated using Recall@k (k = 1, 3, 5, 10, 15). Monolingual retrieval performs poorly (Recall@15<10%), whereas all CLIR approaches substantially improve retrieval accuracy. Among them, BGE-M3 achieves the highest Recall@15, reaching 96.2% for Sinhala-English and 95.6% for Tamil-English, outperforming the best QT approach (Google Translate: 92.4% and 93.0%) while avoiding translation overhead. These results demonstrate that multilingual embedding models provide a more effective and scalable solution for cross-lingual retrieval-augmented generation (RAG) in low-resource government domains.
Cross-lingual information retrieval (CLIR) for low-resource regional languages remains challenging due to limited annotated training data, morphological complexity, and linguistic diversity. This paper systematically investigates Indonesian–Javanese CLIR by integrating traditional sparse lexical retrieval, multilingual dense retrieval, and advanced large language model (LLM)-based reranking within a multi-stage retrieval framework. To evaluate our approach, comprehensive experiments are conducted on the Indonesian–Javanese subset of the CLIRMatrix benchmark, comprising 20,000 Javanese documents, 1,000 Indonesian queries, and 11,000 graded relevance judgments. We compare BM25 with query and document translation, multilingual bi-encoder retrieval, and various LLM-based pointwise reranking methods. The experimental results show that BM25 combined with document translation provides the strongest initial baseline, achieving a Mean Average Precision (MAP) of 0.45, Mean Reciprocal Rank (MRR) of 0.71, and nDCG@10 of 0.55. While LLM-based reranking consistently improves weaker dense retrieval baselines, it often degrades search performance when applied directly on top of strong BM25-based retrieval. To address this, Reciprocal Rank Fusion (RRF) is introduced, which mitigates this degradation issue and achieves the best overall performance, reaching a MAP of 0.47, MRR of 0.74, and nDCG@10 of 0.57. These empirical findings highlight the critical importance of robust initial retrieval baselines and demonstrate that LLM-based reranking is most effective when combined with rank fusion strategies in low-resource CLIR settings.
Raden Mohamad Adrian Ramadhan Hendar Wibawa, Ika Alfina, Evi Yulianti· International Conference on...· 0 citations
In today’s globalized and digitally connected world, individuals increasingly share emotions, opinions, and experiences across multiple languages, making accurate translation essential for cross-lingual sentiment analysis. Although machine translation (MT) is widely used in multilingual applications, the relationships among translation quality, semantic similarity, and sentiment consistency remain insufficiently understood. This study investigates the performance of six LLM-based systems (GPT-4o-mini, Gemini 2.5 Flash-Lite, Qwen 2.5, Llama 3.1, Mistral 7B, and NiuTrans LMT) and four NMT-based systems (Google Translate, Microsoft Translator, NLLB-200, and LibreTranslate-v1.5) in maintaining classifier-mediated sentiment consistency across twelve translation directions involving English, Spanish, French, and Chinese. Experiments were conducted on the Multilingual Amazon Reviews Corpus (MARC), comprising 84,000 randomly sampled user reviews. A multidimensional evaluation framework was used, combining sentiment-consistency metrics (accuracy, weighted F1, MCC, and SSR), translation-quality estimation (COMET-QE), and semantic-similarity assessment (LaBSE). Statistical significance was examined using the Friedman and Nemenyi post hoc tests. The results show that GPT-4o-mini, Gemini 2.5 Flash-Lite, and Google Translate consistently ranked among the strongest systems across multiple evaluation dimensions. Performance differences were particularly pronounced in translation directions involving Chinese, highlighting the influence of language-specific structural characteristics. Furthermore, semantic similarity and translation quality exhibited only moderate relationships with sentiment consistency, indicating that high semantic similarity does not necessarily guarantee strong sentiment consistency. Overall, the findings demonstrate the importance of multidimensional and statistically grounded evaluation frameworks for assessing cross-lingual sentiment consistency and provide practical insights into the strengths and limitations of contemporary MT systems.
E. Cetin, Çağrı Şahin· Applied Sciences· 0 citations
Multilingual sentence embeddings are increasingly used to estimate semantic similarity across languages, yet their sensitivity to fine-grained translation errors remains insufficiently understood. This study investigates whether general-purpose multilingual embedding models can distinguish correct English-Greek translations from minimally modified erroneous alternatives. A contrastive dataset was developed from FLORES+ sentence-aligned reference translations and reviewed by two translation experts. It contains 1,850 examples across ten core and five exploratory error categories, covering factual, lexical-semantic, grammatical, relational, referential, and discourse-level phenomena. Five multilingual sentence-embedding models (BGE-M3, Multilingual E5, Multilingual MPNet, LaBSE, and Jina Embeddings v3) were evaluated using cosine similarity between each English source sentence and its correct and erroneous Greek translations. A reference-free COMETKiwi model was also evaluated as an MT quality-estimation baseline. Performance was assessed through contrastive accuracy and score margins for category-specific sensitivity. BGE-M3 achieved the highest accuracy among embedding models at 89.30 percent, while COMETKiwi achieved 94.49 percent. Embedding models detected explicit factual and lexical changes more reliably than tense-and-aspect and pronoun-coreference errors. COMETKiwi improved performance on several difficult categories, including tense and aspect, pronoun and coreference, and semantic-role errors, but showed lower sensitivity to date-and-time errors and underperformed the embedding models on numbers. The results show complementary error-sensitivity profiles: multilingual sentence embeddings provide useful semantic adequacy signals but are better suited as components of broader translation-evaluation frameworks than as standalone metrics.
Eleftherios Kalogeros, Athanasios Ntalakas, M. Gergatsoulis et al.· 0 citations
DocuMind is a fully offline, privacy-preserving, multilingual Document Question Answering system built on the Retrieval-Augmented Generation (RAG) architecture, enabling true cross-lingual retrieval without any translation step.
S. S, Sripalreddy· International Scientific Jou...· 0 citations
External evidence can make retrieval-augmented generation (RAG) more informative, yet retrieved passages also provide a path for adversarial instructions to enter the model context. We examine that path in an English-Indonesian RAG system and track cross-lingual indirect prompt injection separately at retrieval, reranking, and generation. The experiment starts from 75 semantic items and evaluates every item in eight query/body/payload language combinations, for 600 paired trials. Qwen3-Embedding-0.6B and BGE-M3 produce top-20 candidate sets, BGE-reranker-v2-m3 reduces each set to five documents, and Qwen3-0.6B answers from the resulting context with either a standard prompt or an explicit trust-boundary prompt. Statistical uncertainty is estimated by resampling semantic items, and paired binary outcomes are modeled with generalized estimating equations. Poison documents reached the top 20 in 80.50% of Qwen trials and 37.67% of BGE-M3 trials (odds ratio 6.94, 95% CI 4.53-10.63). Top-five exposure was 18.50% and 16.17%, respectively. Standard end-to-end attack success was 6.33% for Qwen and 6.00% for BGE-M3; boundary-aware prompting lowered both rates to 1.33%, with no canary false positives. The results indicate that multilingual RAG security depends on several linked stages rather than generation alone.
Unknown authors· Journal of Computational Sci...· 0 citations
The scarcity of high-quality semantic lexical resources for Arabic academic text represents a critical bottleneck for natural language processing (NLP) applications, including query expansion, document retrieval, and information organization. This paper presents a framework for inducing semantic neighbor lexicons from the Arabic Research Papers Dataset (ARPD), a publicly available corpus of 2,011 Arabic academic documents spanning seven scientific domains. We exploit type-level representations derived from AraBERT to compute dense, L2-normalized word embeddings and apply GPU-accelerated cosine similarity search to retrieve up to three semantically induced neighbors per vocabulary item. Experiments are conducted on both the raw and preprocessed versions of ARPD. The raw-corpus lexicon covers 168,866 unique terms, and the preprocessed-corpus lexicon covers 159,364, both with 100% three-neighbor coverage and a mean top-1 cosine similarity of 0.91. For document clustering, replacing TF-IDF bag-of-words with AraBERT document embeddings raises the Silhouette coefficient from 0.070 to 0.645 on the raw corpus (+ 820.0%) and to 0.755 on the preprocessed corpus (+ 963.4%), alongside substantial reductions in the Davies–Bouldin index. A pilot expansion experiment under class-based relevance finds no retrieval gain from lexicon expansion, which we report alongside the root-level and expert analyses that explain it. For classification, TF-IDF with only light normalization achieves 99.17% accuracy, exceeding the published benchmark of 99.00% that required heavy preprocessing. Comprehensive comparisons against static embedding baselines (Word2Vec, FastText, GloVe) under both corpus conditions quantify how preprocessing depth interacts with representation type. A comparison against MARBERTv2, AraELECTRA, and CAMeLBERT-MSA under leakage-free, validation-based checkpoint selection shows MARBERTv2 attaining the highest accuracy (99.67% raw, 99.50% preprocessed); we emphasize that differences among the top-performing methods correspond to only a few documents and are not statistically significant, while semantic-neighbor augmentation yields a nominal + 1.82 pp gain for AraBERT on the preprocessed corpus (McNemar $p=0.027$ ).
Ahmad Al Smadi, Yasser Allaham, Lara Rbabah· IEEE Access· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
