Reproducibility in the era of large language models

The Unreasonable Benchmark

Do humans and large language models agree on the quality of synthesis plans?

This work evaluated the agreement between LLMs and expert chemists on holistic evaluations of the retrosynthetic routes and found no agreement on holistic evaluations of the retrosynthetic routes.

Automatic evaluation of assessments using large language models

How to build reproducibility into scientific publishing

The academic publishing system needs to include replication of key experiments wherever possible to meet the replication crisis. A major challenge is to incentive validation and provide funding for it.

Notation matters: cross-representation inconsistency in chemistry language models and its mechanistic origins

Large language models (LLMs) are increasingly applied to molecular property prediction, but whether they encode notation-agnostic molecular representations or exploit surface-level string patterns re- mains untested. We present a dual-phase...

World Journal of Methodology