2026· SemEval@ACL· pp. 899-904· 1 citation· 12 references
Computer Science
TL;DR
This study presents a LLaMA-based approach to answer short cultural questions in different languages within Task 7 of SemEval-2026 (Track 1: SAQ), without access to official training data, and demonstrates that contextual retrieval improves the evaluation of multilingual cultural knowledge.
Abstract
In multilingual and multicultural contexts, LLMs require contextualization mechanisms to generate culturally coherent responses. In this sense, this study presents a LLaMA-based approach to answer short cultural questions in different languages within Task 7 of SemEval-2026 (Track 1: SAQ), without access to official training data. The system integrates controlled synthetic data generation, evidence retrieval through web snippets, and a Retrieval-Augmented Generation (RAG) framework with Few-shot learning. BLEnD is used solely as a thematic guide, ensuring semantic independence. During development, the LLaMA-3.1-8B model achieved 38.51% global accuracy, while LLaMA-3.2-1B obtained 15.54%. In a large-scale evaluation (30,500 instances), the 1B model achieved 16.69% and maintained stability after prompt optimization. The results demonstrate that contextual retrieval improves the evaluation of multilingual cultural knowledge and highlight the importance of pipeline design and model capacity.
Cross-lingual information retrieval (CLIR) for low-resource regional languages remains challenging due to limited annotated training data, morphological complexity, and linguistic diversity. This paper systematically investigates Indonesian–Javanese CLIR by integrating traditional sparse lexical retrieval, multilingual dense retrieval, and advanced large language model (LLM)-based reranking within a multi-stage retrieval framework. To evaluate our approach, comprehensive experiments are conducted on the Indonesian–Javanese subset of the CLIRMatrix benchmark, comprising 20,000 Javanese documents, 1,000 Indonesian queries, and 11,000 graded relevance judgments. We compare BM25 with query and document translation, multilingual bi-encoder retrieval, and various LLM-based pointwise reranking methods. The experimental results show that BM25 combined with document translation provides the strongest initial baseline, achieving a Mean Average Precision (MAP) of 0.45, Mean Reciprocal Rank (MRR) of 0.71, and nDCG@10 of 0.55. While LLM-based reranking consistently improves weaker dense retrieval baselines, it often degrades search performance when applied directly on top of strong BM25-based retrieval. To address this, Reciprocal Rank Fusion (RRF) is introduced, which mitigates this degradation issue and achieves the best overall performance, reaching a MAP of 0.47, MRR of 0.74, and nDCG@10 of 0.57. These empirical findings highlight the critical importance of robust initial retrieval baselines and demonstrate that LLM-based reranking is most effective when combined with rank fusion strategies in low-resource CLIR settings.
Raden Mohamad Adrian Ramadhan Hendar Wibawa, Ika Alfina, Evi Yulianti· International Conference on...· 0 citations
The results show that the RAG architecture provides a scalable alternative for creating precise, contextually grounded conversational agents, thereby mitigating some of the main drawbacks of LLMs.
Rabia Shabbir, K. Talpur, Shakeel Ahmad· ICCK Transactions on Machine...· 0 citations
It is demonstrated that multilingual embedding models provide a more effective and scalable solution for cross-lingual retrieval-augmented generation (RAG) in low-resource government domains.
We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.
Rinit Jain, Tirthraj Mahajan, Advait Joshi et al.· 0 citations
Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. BavGround contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. We evaluate fifteen 7B-10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. We further show that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. BavGround supports localized, protocol-aware evaluation of cultural representation in LLMs.
Jophin John, Michael Hoffmann, Jan Fillies et al.· 0 citations
FinMMEval 2026 Task 1 evaluates multilingual financial multiple-choice question answering in English, Chinese, Arabic, and Hindi. The task tests whether systems can select the correct answer to finance questions involving domain terminology, numerical interpretation, and conceptual financial reasoning across languages and scripts. The final-test set contains 800 questions, with 200 questions per language; gold answers were withheld during submission, and each language was ranked independently by accuracy. The final leaderboards contain 13 English, 11 Chinese, 11 Arabic, and 10 Hindi ranked submissions. Top accuracies range from 92.0% in Hindi to 97.5% in English and Arabic, with the same leading teams appearing near the top across all four languages. The documented systems used retrieval augmentation, direct answer-option scoring, language-specific prompting, selective self-consistency, confidence checks, and LLM-based review stages.
Zhuohan Xie, Yu-Yang Dai, R. Elbadry et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
