2026· Annual Meeting of the Association for Computational Linguistics· pp. 33981-33999· 1 citation· 24 references
Computer Science
Abstract
We introduce MMSciCode, a comprehensive expert-level, multilingual multi-discipline benchmark for evaluating foundation models in scientific code generation. It includes 624 expert-annotated research coding problems spanning six core scientific disciplines. Compared to prior benchmarks, MMSciCode features three key advancements. First, it challenges models to integrate domain-specific knowledge with algorithmic reasoning to implement core functions from research papers. Second, each problem is meticulously annotated by domain experts through a rigorous paper-grounded process, with strict quality controls implemented to ensure dataset integrity and authenticity. Finally, each problem is equipped with comprehensive unit test suites and con-tainerized environments, enabling reproducible and diagnostic evaluation of both functional correctness and domain validity. We conduct an extensive evaluation of 23 state-of-the-art foundation models and 2 coding agents on MMSciCode. We identify substantial performance gaps between models and human experts, providing actionable insights for advancing expert-level scientific code generation.
Pseudo2Code, a benchmark designed to systematically evaluate the impact of structured pseudocode on code generation quality and algorithmic faithfulness, is introduced and the Pseudo2Code Agentic Framework is proposed, a multi-stage pipeline that leverages pseudocode as an explicit intermediate reasoning representation...
Shadikur Rahman, Umme Ayman Koana, S. Danish· 0 citations
Benchmark evaluations reveal that agent performance varies substantially across languages and drops sharply on the harder cross-lingual tasks, and analysis shows that multilingual execution exposes systematic failure modes across planning, tool interaction, and decision-making in long-horizon agents.
Hongliang Li, Yijin Liu, Zhiwei Zhang et al.· 0 citations
We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational cur...
Rinit Jain, Tirthraj Mahajan, Advait Joshi et al.· 0 citations
Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge. In nuclear engineering, problem solving often requires not only factual knowledge but also quantitative reasoning and conceptual und...
Henry Yuchi, Michal Kucer, Benjamin H. Sims et al.· arXiv.org· 0 citations
Although LLMs significantly advance fact-checking, structural challenges persist, primarily hallucinations, data scarcity, data scarcity, and a lack of temporal awareness, and a research agenda prioritizing temporal-aware architectures, Small Language Models for sustainable deployment, and human-centered evaluation met...
Luis Henrique M. Carvalho, R. Goldschmidt· IEEE Access· 0 citations
Compilable Academic Document Parsing (CADP) is proposed, a paradigm that reconstructs a full page as contextual \LaTeX{} plus executable Python, so that structure-preserving elements and executable chart representations can be reconstructed, recompiled, and directly verified against the source page.
Rihui Jin, Jun Wang, Chen Zhu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
