Skip to content

MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?

Sep 2026 · 1 citation · 38 references
Computer Science

TL;DR

MechBench is introduced, a benchmark that explicitly separates mechanism discovery from phenomenal-law recovery and reveals a substantial generalization gap in mechanistic reasoning, establishing mechanism discovery as a distinct challenge beyond recovering observable scientific laws.

Abstract

Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them. Existing benchmarks for symbolic regression and scientific agents primarily evaluate phenomenal-law recovery, leaving mechanism discovery largely untested. We introduce MechBench, a benchmark that explicitly separates these two capabilities. Each task is defined by a mechanistic model, a structured set of scientifically meaningful relations whose joint consequences entail an observable phenomenal law, while agents receive only observational data and scientific context. We evaluate mechanism recovery through mechanism probes, which query internal scientific consequences that cannot be inferred from the phenomenal law alone. To reduce reliance on memorized textbook mechanisms, we construct unfamiliar variants through controlled, scientifically interpretable mutations of canonical mechanisms, and screen for mechanistic indistinguishability to exclude ambiguous instances admitting comparable competing mechanisms. Experiments across representative scientific agents reveal a substantial phenomenal--mechanism recovery gap: for Codex with GPT-5.6-sol, phenomenal-law accuracy reaches 35.00% on the Core-set while mechanism accuracy is only 13.75%, with mechanism recovery failing in 64.29% of cases where the phenomenal law is correctly recovered. The gap widens as mechanisms become increasingly mutated, and even providing the correct phenomenal law leaves mechanism recovery below 50%. These results reveal a substantial generalization gap in mechanistic reasoning and establish mechanism discovery as a distinct challenge beyond recovering observable scientific laws.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Can AI Scientists Change Their Minds? Prior-Evidence Conflict in Synthetic Universes

Synthetic Universes is introduced, a controlled benchmark that pairs canonical famous worlds with matched twisted twins governed by nearby noncanonical mechanisms and establishes a narrower verification result: predictive adequacy and mechanism recovery are distinct scientific claims and require distinct tests.

Kargi Chauhan · 0 citations
#artificial intelligence Preprint Sep 2026

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

It is found that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and the best-of-N study reveals a selection bottleneck.

Yi-Ming Huang, Zi-Chen Liu, Jun-Xia Cui et al. · 2 citations
#artificial intelligence Preprint Oct 2026

From Scientific Observations to Mechanisms: Benchmarking Hypothesis Generation by AI Scientists

MechHypoBench is introduced, the first benchmark for evaluating whether AI agents and AI scientists can generate mechanistic hypotheses from empirical data, and an evaluation framework is developed that assesses open-form mechanistic hypotheses through their consequences under withheld conditions.

Xia-Xun Xie, Qing-Qing Long, Meng Xiao et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Science sandboxes measure the scientific capability of AI agents

This work instantiates science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision, in two biological settings, models of regulatory genomics and protein fitness prediction, and examines the capabilities of frontier agents.

Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai et al. · 1 citation
#machine learning Preprint Sep 2026

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era

CausalArena is introduced, a unified and evolvable benchmark for causal discovery under a common protocol, and substantial ranking shifts across SCM families and protocols are revealed, showing that strong performance in one benchmark regime does not reliably transfer to others.

Zi-Rong Li, Si-Zhuang Liu, Tian-Zuo Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Explore, Then Commit: Measurement-Efficient Scientific Law Discovery with Language Models

An explore-then-commit protocol in which a large language model proposes hypotheses, a programmatic planner gathers measurements, and a fresh prompt synthesizes the final law from fixed observations is evaluated, supporting the complete protocol as a promising measurement-efficient configuration.

Kautik Mandve, Dileepa Fernando · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.