Skip to content

A Benchmark Framework for Screening Automation in Systematic Reviews

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

An evaluation framework that accounts for class imbalance is proposed, i.e., the natural prevalence of excluded articles relative to included articles in SRs, and PromptSR, a tool designed to support prompt experimentation, experiment management, and result analysis for LLM-based screening are introduced.

Abstract

Systematic reviews (SR) are essential for evidence-based research, but their screening phase is highly time-consuming and labor-intensive. Large language models (LLMs) offer a promising opportunity to reduce this workload by assisting with article relevance classification. However, existing evaluation approaches often rely on traditional metrics that may be misleading for highly imbalanced SR screening datasets. This paper presents a benchmark dataset of $45\,064$ labeled entries for evaluating LLM performance in SR screening across 32 curated secondary studies. It proposes an evaluation framework that accounts for class imbalance, i.e., the natural prevalence of excluded articles relative to included articles in SRs. It also introduces PromptSR, a tool designed to support prompt experimentation, experiment management, and result analysis for LLM-based screening. We also present a use case demonstrating the application of SRBench and PromptSR.

View source

Similar papers

Review Open access 2026

Large Language Model-Based Automated Assessment: A Systematic Review, Taxonomy, and Implications for Personalized Learning

Research on large language model (LLM)-based automated assessment (AA) has expanded rapidly. Nevertheless, the literature remains fragmented across contributions, models, implementation configurations, datasets, and evaluation metrics, complicating efforts to identify approaches suitable for personalized learning. This...

H. D. Septama, A. Permanasari, R. Ferdiana · 0 citations

PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

This work analyzes SciLitBench, a corpus of 888 review-automation papers with 14,726 annotations, to characterize changes in methods, review-stage use, evaluation and reported limitations, and introduces PRISMA-LLM, an empirically grounded framework separating implementation disclosure from consequence-sensitive evalua...

Miguel Zabaleta, Bai-Han Lin · 0 citations
#large language models Review Sep 2026

Large language models for literature screening in conceptually complex and interdisciplinary reviews

It is suggested that current LLMs can support literature screening, but their reliability depends strongly on the conceptual clarity of the review task and the structure of eligibility criteria, as well as practical implications for the transparent and responsible use of LLMs in informetric research and systematic revi...

Ni Cheng, Heng Dong, Xuan Han et al. · 0 citations
Review Open access Aug 2026

Artificial Intelligence Resources for the Screening of Titles and Abstracts in Systematic Reviews: A Scoping Review

This review aims to identify current evidence concerning AI use during preliminary SLR reference screening and describes characteristics such as the different metrics used for reporting performance and how the different algorithms, pipelines, workflows or web applications are validated.

A. M. Barragán, Sara Elena Ortiz Bonett, Eliana-Isabel Rodríguez-Grande et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.