CLINES: Clinical LLM-based Information Extraction and Structuring Agent

Abstract

Abstract Objective Clinical narratives in electronic health records (EHRs) hold diagnostic, therapeutic and temporal information often missing from structured fields; manual chart review remains the de facto standard but is slow, costly and variable. We aimed to develop and evaluate Clinical LLM‑based Information Extraction and Structuring Agent (CLINES), a pipeline that extracts and normalises clinical concepts from free-text notes into ontology-grounded, auditable, schema-ready data without task-specific fine-tuning. Methods and analysis We developed CLINES, a modular multistep pipeline: semantic chunking of long notes; extraction by reasoning-capable large language models (LLM); attribute assignment (assertion, value+unit); Unified Medical Language System normalisation; explicit and relative date resolution; and aggregation into an i2b2-style schema. Evaluation was conducted without task-specific supervised training on deidentified MIMIC-III, 4CE and CORAL (breast, pancreas) notes. Comparators include rule/lexicon systems, transformer encoders, single-prompt LLM baselines, alongside chain-of-thought LLMs and full-size clinical named entity recognition encoders (BERT-base, GatorTron-base). Outcomes were F1 for entity, assertion, value+unit and date processing, reported with 95% bootstrap CIs and paired-bootstrap, false discovery rate (FDR)-corrected tests. Inter-annotator agreement, hallucination taxonomy, cost and component ablations were also assessed. Results Across all datasets and tasks, CLINES ranked first on every task-dataset combination. F1 ranges were 0.69–0.87 (entity), 0.84–0.93 (assertion), 0.77–0.90 (value+unit) and 0.73–0.79 (date, where gold annotations were available). Gains over the strongest single-prompt LLM were +0.21–0.38; transformer encoders trailed by +0.28–0.68 F1 on entity extraction. Performance remained stable as notes lengthened, while transformer baselines lost recall. Code field gains and most assertion/value comparisons were significant at FDR-adjusted p<0.05 on paired-bootstrap tests; mention-level inter-annotator F1 on a cross-annotated subset was 0.82 (assertion κ=0.85); a seven-class taxonomy on 700 stratified false positives found 54% of apparent false positives to be annotation gaps rather than true hallucinations. Conclusion CLINES translates narrative EHR text into ontology-grounded, auditable, schema-ready data, offering a practical route to scale chart review-like extraction for cohort discovery and real-world evidence. CLINES substantially reduces but does not eliminate the need for manual review: the residual hallucination quantified here means human verification remains necessary for safety-critical use. CLINES is model agnostic—open and closed models can be substituted to meet cost, performance and privacy goals.