We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.
Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an evaluation blind spot concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce MORE, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) Unprecedented Scale: It covers 149 languages, making it the most linguistically diverse benchmark to date; (2) Structural Complexity: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) Data Authenticity: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark's effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.
Long Xu, Binghong Wu, Ting-Ting Yu et al.· 3 citations
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
Daryna Dementieva, N. Babakov, Kathy Hammerl et al.· 0 citations
Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual understanding. To address the need for systematic evaluation in this domain, we introduce NuclearQAv2, a benchmark for assessing LLMs on nuclear engineering knowledge. The benchmark comprises approximately 1,240 question-answer pairs spanning three categories: boolean, numeric, and verbal. NuclearQAv2 is constructed using a hybrid pipeline that combines expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific technical corpora. By leveraging structured prompting for both automated question generation and response evaluation, the proposed framework enables scalable benchmark construction and evaluation. We evaluate a diverse set of LLMs using NuclearQAv2 and observe substantial performance differences across task types. While the models generally perform well on factual questions, quantitative reasoning and conceptual understanding remain considerably more challenging. These results highlight the importance of multi-faceted evaluation frameworks and establish NuclearQAv2 as a scalable benchmark for assessing LLM capabilities in technical domains.
Henry Yuchi, Michal Kucer, Benjamin H. Sims et al.· arXiv.org· 0 citations
Large language models (LLMs) have achieved remarkable success in high-resource languages, yet their performance on Traditional Mongolian remains highly limited. A primary bottleneck is the absence of a systematic evaluation framework, which precludes quantitative comparison and obscures directions for model optimization. In this paper, we introduce TM-Bench, the first comprehensive benchmark for LLMs on Traditional Mongolian. TM-Bench adopts a hybrid construction strategy consisting of human-verified Translation-based Adaptation, Expert-Original Authoring, and Semi-automated Synthesis. It comprises 18,357 instances spanning five tasks across both natural language understanding and generation to evaluate models' reasoning, knowledge application, and linguistic proficiency. We conduct systematic evaluations across representative model families. The results show that on understanding tasks, model performance lags significantly behind high-resource languages, with only a few models performing slightly above the random baseline. For generation tasks, both automatic metrics and double-blind human evaluations reveal severe semantic collapse, failing to generate coherent text and often producing unreadable gibberish. These findings underscore the critical role of TM-Bench as a foundational infrastructure for evaluating LLMs in Traditional Mongolian and catalyzing future model optimization. Our benchmark and code are available at https://github.com/gao1948083886/TM-Bench.
Zhenjie Gao, Feilong Bao, Aruukhan Bai et al.· Annual International ACM SIG...· 0 citations
We introduce Grounded Multilingual Task Worlds for Romanian (GMTW-Ro), a benchmark designed to evaluate whether large language models can reliably follow complex instructions in Romanian, rather than merely produce fluent text. Existing Romanian benchmarks largely rely on multiple-choice formats, answer extraction, or model-based evaluation, which struggle to assess multi-constraint reasoning and structured task completion. GMTW-Ro addresses these limitations through grounded task worlds: fully specified environments in which model outputs are verified via deterministic, programmatic checks. The benchmark spans four task domains—travel planning, calendar scheduling, context-grounded question answering, and dietary menu planning—requiring both a structured JSON plan and a natural-language explanation in Romanian. Evaluation is decomposed into three orthogonal metrics: Understanding (U), measuring constraint adherence and instruction-following; Generation (G), assessing Romanian text quality through diacritic accuracy, language purity, and code-switching absence; and Faithfulness (F), quantifying consistency between generated plans and their explanations. All instances are automatically verified as solvable using backtracking algorithms. We release two curated datasets: a standard benchmark of 500 instances and an adversarial set of 300 instances with heightened constraint complexity, alongside the complete evaluation toolkit and a purpose-built Romanian NLP library. Evaluation of 11 models reveals substantial performance variation (58.6%–90.7%) and exposes a pronounced knowledge–behavior gap, where models with fluent Romanian generation nevertheless fail core reasoning tasks. Most notably, Romanian-finetuned models underperform their base counterparts: RoLlama3.1-8B scores 20.1 percentage points below Llama-3.1-8B, with structured JSON output success dropping from 95 to 44%. These results raise important questions about how current language adaptation pipelines preserve instruction-following and structured reasoning capabilities.
Andrei-Ștefan Bulzan, Bogdan Morariu, Andrei-Razvan Joldea et al.· Frontiers in Artificial Inte...· 0 citations
Multilingual Question Answering (MQA) is primarily characterized by semantic-based and symbolic-based approaches, remain constrained by Language Confinement. Semantic methods, such as RAG and Agentic frameworks, suffer from linguistic bias that restricts retrieval and reasoning to the source language, while symbolic approaches struggle with cross-lingual schema alignment. We consider that integrating multilingual information from the semantic space into the symbolic space to formulate precise queries enables the effective utilization of global multilingual knowledge. In this paper, we propose SeSyCo, a Semantic-Symbolic Knowledge Consensus framework. Specifically, we leverage the semantic space to diverge monolingual queries into broad multilingual evidence, and subsequently utilize the symbolic space to eliminate language discrepancies, converging the gathered information into a robust consensus for precise SPARQL generation. Extensive experiments on the MLaKE dataset demonstrate that SeSyCo outperforms the strongest baseline by 10.7% in multi-hop settings, validating that establishing a robust multilingual consensus is essential for enhancing MQA performance. The collection is available at https://github.com/YuZhang9408/SeSyCo.
Yu Zhang, Ran Song, Xiaofei Gao et al.· Proceedings of the 32nd ACM...· 0 citations