Automated literature screening using large language models in headache research: a comparative performance evaluation of seven models using zero-shot and iterative prompt refinement
Abstract Background Large language models (LLMs) show promise for automating literature screening, although their performance varies across clinical domains and further evaluation is required for broader implementation. This study aimed to assess the accuracy, reliability and feasibility of literature screening in headache research across seven LLMs, spanning compact to frontier models (GPT-4o, DeepSeek-V3, and Meta’s LLaMA-3 at multiple scales). Methods This study is a comparative methodological evaluation of LLM-assisted title and abstract screening. Using a dataset from a real-world systematic review including randomized controlled trials on placebo and nocebo responses in migraine, we evaluated classification performance, processing runtimes, and computational costs across seven models using zero-shot prompting and IMAPR (Iterative Multi-Agent Prompt Refinement). The IMAPR system is our in-house developed, LLM-driven multi-agent framework in which a single, fixed LLM iteratively refines prompts. Each model was run five times per method, and sensitivity was assessed against a predefined threshold of ≥ 95%. Results Zero-shot sensitivity means ranged from 95.1 to 100%, with all models except GPT-4o meeting the ≥ 95% threshold in every run, while specificity ranged from 9.0 to 89.9%. IMAPR increased GPT-4o sensitivity above 95% in all runs, with a minor specificity decrease (-0.9%), while sensitivity became more variable in other models. Specificity improved significantly for LLaMA-3.1 8B (59.2% to 82.4%) and DeepSeek-V3 (77.8% to 85.8%), reducing screening workload at the mean cost of two to three fewer identified relevant citations. Positive predictive value improved but remained low (3.4–23.3%). Zero-shot screening completed within 50 min per model at a cost of up to $4.54, and the complete IMAPR workflow within three hours at a cost below $11. Conclusion This study suggests that LLM-based automated literature screening could be reliable, fast, and affordable in headache research. We encourage researchers and clinicians to critically and prospectively evaluate LLM-assisted screening across diverse medical specialties, independent datasets, and future generations of LLMs to facilitate responsible broader adoption. Trial registration Not applicable.