MechBench is introduced, a benchmark that explicitly separates mechanism discovery from phenomenal-law recovery and reveals a substantial generalization gap in mechanistic reasoning, establishing mechanism discovery as a distinct challenge beyond recovering observable scientific laws.
Abstract
Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them. Existing benchmarks for symbolic regression and scientific agents primarily evaluate phenomenal-law recovery, leaving mechanism discovery largely untested. We introduce MechBench, a benchmark that explicitly separates these two capabilities. Each task is defined by a mechanistic model, a structured set of scientifically meaningful relations whose joint consequences entail an observable phenomenal law, while agents receive only observational data and scientific context. We evaluate mechanism recovery through mechanism probes, which query internal scientific consequences that cannot be inferred from the phenomenal law alone. To reduce reliance on memorized textbook mechanisms, we construct unfamiliar variants through controlled, scientifically interpretable mutations of canonical mechanisms, and screen for mechanistic indistinguishability to exclude ambiguous instances admitting comparable competing mechanisms. Experiments across representative scientific agents reveal a substantial phenomenal--mechanism recovery gap: for Codex with GPT-5.6-sol, phenomenal-law accuracy reaches 35.00% on the Core-set while mechanism accuracy is only 13.75%, with mechanism recovery failing in 64.29% of cases where the phenomenal law is correctly recovered. The gap widens as mechanisms become increasingly mutated, and even providing the correct phenomenal law leaves mechanism recovery below 50%. These results reveal a substantial generalization gap in mechanistic reasoning and establish mechanism discovery as a distinct challenge beyond recovering observable scientific laws.
Synthetic Universes is introduced, a controlled benchmark that pairs canonical famous worlds with matched twisted twins governed by nearby noncanonical mechanisms and establishes a narrower verification result: predictive adequacy and mechanism recovery are distinct scientific claims and require distinct tests.
It is found that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and the best-of-N study reveals a selection bottleneck.
Yi-Ming Huang, Zi-Chen Liu, Jun-Xia Cui et al.· 2 citations
MechHypoBench is introduced, the first benchmark for evaluating whether AI agents and AI scientists can generate mechanistic hypotheses from empirical data, and an evaluation framework is developed that assesses open-form mechanistic hypotheses through their consequences under withheld conditions.
Xia-Xun Xie, Qing-Qing Long, Meng Xiao et al.· 0 citations
This work instantiates science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision, in two biological settings, models of regulatory genomics and protein fitness prediction, and examines the capabilities of frontier agents.
Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai et al.· 1 citation
CausalArena is introduced, a unified and evolvable benchmark for causal discovery under a common protocol, and substantial ranking shifts across SCM families and protocols are revealed, showing that strong performance in one benchmark regime does not reliably transfer to others.
Zi-Rong Li, Si-Zhuang Liu, Tian-Zuo Wang et al.· 0 citations
An explore-then-commit protocol in which a large language model proposes hypotheses, a programmatic planner gathers measurements, and a fresh prompt synthesizes the final law from fixed observations is evaluated, supporting the complete protocol as a promising measurement-efficient configuration.
Kautik Mandve, Dileepa Fernando· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.