Jul 2026· Journal of Epidemiology· 0 citations· 19 references
Medicine
TL;DR
LLM-assisted screening may improve efficiency without compromising reliability when accompanied by appropriate safeguards for ambiguous records, and many model-human disagreements reflect topic- and workflow-dependent screening conventions rather than intrinsic model failure.
Abstract
Background
We evaluated the reliability of large language models (LLMs) for abstract screening under real-world review practices and qualitatively characterized model-human discordance to inform safe workflow integration.
Methods
We evaluated GPT-4.0, GPT-5.0, and GPT-5.0-mini on two curated systematic review datasets representing contrasting topic densities, defined by the target-to-background ratio (TBR): a core-subject dataset (TBR 69%) in which the target intervention was central, and a peripheral-subject dataset (TBR 4.4%) in which the target intervention was incidental. Final inclusion after full-text review served as the reference standard. We developed a qualitative taxonomy of disagreements, classifying false negatives as intended human leniency, gray-zone ambiguity, or true LLM misses, and false positives as implicit or additional human exclusion rules, gray-zone ambiguity, or nominal inclusions that increase workload only.
Results
GPT-5.0-mini achieved the best sensitivity-efficiency trade-off (core-subject: 91% sensitivity with 96.7% workload reduction; peripheral-subject: 83% sensitivity with 92.7% workload reduction) and negative predictive value >99% in both datasets. Disagreement was lower when relevance was central (core-subject: 1.6%, 7/430) with no true LLM misses (0/430). In the peripheral-subject dataset, disagreement was higher (10.6%, 74/696), driven mainly by intended human leniency among false negatives (52/56) and gray-zone ambiguity among false positives (12/18), while true LLM misses remained rare (0.4%, 3/696).
Conclusion
Many model-human disagreements reflect topic- and workflow-dependent screening conventions rather than intrinsic model failure. LLM-assisted screening may improve efficiency without compromising reliability when accompanied by appropriate safeguards for ambiguous records.
Background. Large language models (LLMs) are increasingly used for screening in evidence synthesis, where false negatives can remove relevant studies before full-text assessment. We compared human and LLM title-and-abstract screening workflows in a preregistered study embedded in a conceptually complex scoping review. Methods. After a conservative title-only screen, 1,131 records were screened by one review lead, four trained assistants screening non-overlapping subsets, and seven complete LLM runs using different models and processing configurations, including a nominally identical repeat run. We compared retained workload, operational recall against 316 verified eligible records, agreement, run-to-run consistency, and procedural burden. Because eligibility was verified only for records advanced and assessed in the parent review, recall estimates were operational. Results. No workflow recovered all verified eligible records. The human workflows and two GPT-5.4 file-batch runs retained 42.2-45.0% of records while achieving 82.3-82.9% recall. Gemini 3.1 file batches achieved the highest recall (83.9%) but retained 56.7% of records. All-at-once configurations recovered fewer eligible records than corresponding file-batch configurations. Two nominally identical GPT-5.4 file-batch runs agreed on 91.7% of records but differed on 94 records, including 29 verified eligible records retained by only one run. Discussion. LLM screening performance depended on the implemented workflow, not model identity alone. Processing configuration, workload, record-level variation, and human-LLM decision integration are therefore substantive properties of deployed systems. For high-recall tasks, LLMs are better suited to validated, auditable, human-supervised workflows than autonomous exclusion.
Nikol Figalová, Lynn Huestegge, Anne Böckler-Raettig· 0 citations
OBJECTIVE
To evaluate the performance and consistency of Large Language Models (LLMs) in core systematic review (SR) tasks and to introduce open-source tools for automated batch processing that provide decision rationales.
METHODS
We assessed GPT-4o, Kimi-K2, DeepSeek-V3, and DeepSeek-R1 on five SR tasks: title/abstract screening (3550 records), full-text screening (233 texts), data extraction (112 RCTs), Risk of Bias (ROB) assessment (112 RCTs), and AMSTAR-2 assessment (20 SRs). Each model was evaluated twice to measure consistency. All outputs required supporting rationales and verbatim evidence.
RESULTS
LLMs demonstrated proficiency across tasks, with generally high intra-model but lower inter-model consistency. In screening, models showed lower precision (0.27-0.40) but high recall (0.83-0.91) and specificity (0.83-0.91). DeepSeek-R1 and DeepSeek-V3 excelled in title/abstract and full-text screening, respectively. Data extraction accuracy was similar across models (0.78-0.82). Kimi-K2 achieved the highest ROB F1 score (0.71). AMSTAR-2 assessments were generally acceptable.
DISCUSSION
While effective, LLMs showed variable performance across SR tasks. The mandatory output of rationales and evidence enhances transparency and allows for human verification of AI decisions.
CONCLUSION
We provide a suite of automated tools for key SR tasks. By leveraging these tools to validate model outputs rather than starting manually, reviewers can significantly improve workflow efficiency while maintaining methodological rigour.
Unknown authors· Journal of Evaluation In Cli...· 0 citations
Large language model editorial re--ations were prompt sensitive and showed fair agreement across models despite critique statements that were largely grounded in manuscript text, supporting assistive use with human oversight.
S. Erturk, Mustafa Durmaz· Academic Radiology· 1 citation
Objective: To characterize the kinds of internal documentation inconsistencies a general-domain large language model (LLM) can surface from real-world discharge summaries, and to identify recurring failure modes that limit reliability at scale. Materials and Methods: We applied a two-stage LLM pipeline---open-ended candidate identification (Gemini 2.5 Pro) followed by context-grounded verification (Gemini 2.5 Flash)---to 3,000 randomly sampled MIMIC-IV-Note discharge summaries. A subset of the pipeline output was then reviewed manually by clinical experts. Results: Our pipeline surfaced 3,460 candidate inconsistencies, affecting 69.7% of admissions. Representative examples spanned demographics, allergies, procedures, diagnoses, laboratory, medications, and care-planning domains, with direct implications for clinical reasoning or patient safety. Expert review also revealed recurring failure modes that arise when verification requires temporal reasoning, evolving-diagnosis context, or knowledge of outpatient-prescribing conventions the model does not natively possess. Discussion: Detection is highly context-dependent: many flagged pairs require anchoring each statement to its source section and clinical domain, then assessing whether the conflict reflects a true contradiction or missing context. We propose a graded ontology spanning strict contradiction and ambiguity, with a schema characterizing each flagged case by category, section, domain, and inconsistency axis. Conclusion: This formative study establishes a methodological foundation and conceptual framework to guide subsequent validated, large-scale EHR-inconsistency analysis.
Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening $F_2$ by 28.8\%, while researcher-authored rationales improve full-text screening by 15\%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30\% of annotated evaluation evidence and 25\% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.
Large language models (LLMs) are increasingly used to draft medical manuscripts, yet their citations are unreliable and clinicians lack a validated way to verify them. We evaluated three frontier LLMs, Claude Opus 4.8, GPT-5.5, and Gemini 3.5 Flash, generating 270 cardiology narrative reviews with web search enabled, and verified all 8,050 references against PubMed. Problematic references accounted for 11.5% of GPT-5.5 output, 29.2% of Claude output, and 29.6% of Gemini output (P < 0.001), with no significant gradient across topics of differing publication volume (P = 0.052). Misattribution, a valid PubMed identifier that resolves to a different article, made up 77% of errors, whereas fabrication was rare (0.6%). Against an expert-adjudicated set of 270 references, an LLM-based Chain-of-Verification (CoVe) detected 60 of 62 problematic references (sensitivity 96.8%, specificity 98.6%), including every misattribution and fabrication. LLM-generated citations require identifier-level verification, and CoVe provides it at expert-level accuracy.
Unknown authors· medRxiv· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
