The Ability of Generative Artificial Intelligence to Evaluate the Quality of Scholarly Output

Abstract

Large language models (LLMs) are increasingly being considered as tools to support the peer-review process. However, we currently have a limited understanding of their capabilities and biases in this area. In a set of three studies, we examined how out-of-the box LLMs evaluate scientific quality when presented with controlled, systematically manipulated research abstracts from various scientific disciplines (Study 1 & 2), and Psychology in particular (Study 3). Our design independently varied six key dimensions: Novelty, Analysis Quality, Internal Validity, Sample Size, Preregistration, and Statistical Significance. The results showed that GPT-4o-mini is sensitive to all six variables to some extent, but Statistical Significance had by far the strongest effect: Abstracts that mentioned finding a statistically significant effect received substantially higher ratings compared to abstracts that reported null effects but were otherwise identical. In contrast, newer models with more reasoning capabilities (i.e., GPT-5.6-terra and Claude-sonnet-5) consistently preferred abstracts with higher sample size, analysis quality and internal validity. Abstracts mentioning that the study was preregistered also tended to receive somewhat higher ratings. Interestingly, statistical significance and novelty consistently failed to explain a significant amount of variability in the ratings of those models. Taken together, these results highlight both the potential of advanced LLMs to support scientific evaluation and the importance of understanding the criteria that drive their judgments before incorporating them into peer-review workflows.