Skip to content
Book Open access

Reliable Evaluation of AI Assisted Peer Review

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · pp. 9452-9454 · 0 citations · 26 references

TL;DR

This talk presents case studies in which evaluation metrics reward surface-level improvements like polish and fluency while overlooking evidentiary content, correctness, and editorial usefulness, and discusses practical lessons for building reliable evaluation systems in real peer review workflows.

Abstract

Large language models (LLMs) are rapidly moving from experimental prototypes into operational peer review workflows, where they are used to polish, rewrite, synthesize, and generate reviews. This shift creates an urgent challenge for organizations that depend on expert assessment. Rather than asking only whether AI can improve peer review, this talk confronts a more fundamental question: can we reliably measure what makes a review useful, fair, and decision-relevant? The talk examines this question through the lens of AI-assisted peer review and shows that commonly used signals and LLM-as-a-judge evaluation often fail to capture substantive review quality. Drawing on real-world deployment experience at Reviewerly, we present case studies in which evaluation metrics reward surface-level improvements like polish and fluency while overlooking evidentiary content, correctness, and editorial usefulness. We then discuss practical lessons for building reliable evaluation systems in real peer review workflows. Although peer review is the central case study, the broader message applies to many industry and organizational settings where AI is being introduced into expert judgment workflows. Responsible automation requires more than stronger generation models. It requires evaluation methods that measure the qualities we actually want AI systems to improve.

Read PDF

Similar papers

Review 2026

AI-Assisted Peer Review - A framework for more efficient and reliable scientific evaluation

Peer review is central to scientific quality assurance, yet it faces persistent challenges, including reviewer workload, delays, variability in assessment and potential bias. Artificial intelligence (AI), particularly large language models, may support selected aspects of manuscript evaluation, but its reliability, lim...

S. Kitsinelis · 0 citations

Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three...

Fan Huang, Hong-Yu Zhou · 0 citations

AI Reviews Are Biased Towards the Appendix

Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. I...

Saptarshi Ghosh, Mohammad Saim, Tianyu Jiang · 0 citations
#natural language process... Book Open access Sep 2026

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics

Findings show that many peer-review evaluation metrics partially conflate review quality with linguistic presentation, and indicate that robustness to meaning-preserving rewriting should be validated before such metrics are used to compare human-written, AI-assisted, and AI-generated reviews.

Shakiba Amirshahi, Sajad Ebrahimi, Hai-Son Le et al. · 1 citation

ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judg...

Kihyun Na, Injung Kim · 0 citations
Review Open access Sep 2026

Evaluating large language models as grant reviewers: a comparative study of prompt engineering strategies

Grant application review is resource-intensive and subject to inter-rater variability. Large language models (LLMs) may augment this process, but their reliability in grant evaluation remains unexplored. This exploratory pilot study compared LLM-generated grant reviews to human expert reviews across three prompt engi...

Hants Williams, Jack Evan Lamberg, Eric M. Lamberg · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.