Diagnosing the limits of large language models for structured scientific evidence extraction

Abstract

Scientific evidence is predominantly communicated through unstructured articles, while many downstream workflows require structured representations of study populations, variables, methods, and quantitative findings. Large language models (LLMs) offer a promising route toward automating this transformation, yet their reliability in constructing schema-constrained evidence records from parsed scientific full texts remains insufficiently understood. Evidence synthesis is a representative use case in which systematic reviews and meta-analyses require accurate, structured study information before quantitative integration can begin. We present a schema-grounded, tiered evaluation framework for structured scientific evidence extraction that increases task difficulty from single-entity identification to relational tuples and corpus-level statistical tasks. We evaluate frontier commercial and open-weight models across five scientific domains under a stricter multi-level assessment protocol, complemented by human validation of model-based judging and process-level inspection of intermediate reasoning for statistical tasks. Results reveal a clear pattern: models perform substantially better on low-complexity extraction tasks than on tasks requiring correct bindings among entities, numerical values, and methodological attributes. Intermediate-step analysis further indicates error amplification under exact downstream aggregation, where small upstream errors often prevent correct final outputs. Scaling model size yields uneven gains and does not remove these weaknesses. These findings show that recent advances have not resolved the fundamental challenges of structured scientific evidence extraction. We provide a reusable query suite and diagnostic evaluation framework to support future fine-grained benchmarks and more rigorous assessment of models for evidence-intensive scientific workflows.