Development and Accuracy Evaluation of an LLM-Assisted Workflow for Screening and Data Extraction in Large-Scale Syntheses

Abstract

Large-scale research syntheses are labor-intensive and prone to human error, and large language models (LLMs) could support their screening and data extraction. Building on current recommendations, we developed an integrated workflow for title and abstract screening (Step 1), full-text screening (Step 2), and data extraction (Step 3), and evaluated it in a preregistered proof-of-concept study.The study updated a meta-analysis on the rank-order stability of cognitive abilities, in which records may report several nested coefficients. Every step used ensembles of five LLM-instances, and we evaluated 2,960 workflow configurations against a human gold standard of 892 records, of which 692 records with 102 coded effect sizes formed the evaluation set. Without human review, Steps 1 and 2 retained 96.72% and 100.00% of eligible records. At Step 3, the LLMs missed 35.05% of gold effect sizes in rule-based extracted text but none in text from machine-learning-based parsing, which also lowered their coding error from 10.16% to 2.61%. In the recommended configuration, human review was confined to Step 3, where checking effect sizes of 5 of 40 records and 16.43% of the cells raised specificity to 100.00% and coding accuracy from 97.39% to 99.24%.We derive five recommendations: (1) refine criterion-specific prompts on a calibration set, (2) use machine-learning-based PDF parsing for data extraction, (3) aggregate ensemble ratings leniently at screening and strictly at data extraction, (4) focus human review on data extraction, targeted via flags and ensemble disagreement, and (5) identify nested effect sizes in a separate stage with unique identifiers.