Development and validation of natural language processing pipelines to extract injury and surgery-related data elements from ACL reconstruction operative reports
The NLP pipelines have the potential to extract relevant operative information for a large cohort of patients to support clinical research, such as identifying predictors of subsequent surgery after ACLR and can inform the planning of future studies using larger training sets and more advanced NLP methods.
Abstract
Over 100,000 patients undergo ACLR each year in the United States, providing rich electronic medical record data to improve outcomes, but manual chart review can be time-consuming and expensive. Natural language processing (NLP) methods have been successfully used to extract information for other orthopaedic operative procedures (e.g., hip and knee arthroplasties). Different subspecialities, however, use diverse terminology or “sublanguages” to describe clinical concepts and NLP models perform better concept extraction when trained on specialty-specific texts. Therefore, the purpose of this study was to develop and validate a reliable NLP pipeline to extract meaningful clinical data elements (e.g., graft type, meniscal involvement) from ACLR operative reports. Operative reports for a training and test set were randomly selected based on surgeon volume and year from individuals who underwent ACLR within a single healthcare system between 2013 and 2021. Clinical Language Annotation, Modeling, and Processing Toolkit (CLAMP) was utilized to train a domain-specific model and build a pipeline for data extraction of clinically meaningful data elements, including injury-related factors and surgical factors. Relevancy metrics were calculated using values identified by a single clinician as the gold standard. Overall, the individuals selected for the training and test sets (n = 437 total) were 26.6 ± 10.9 years old, and 43.5% were female, similar to the age (26.7 ± 11.5 years) and proportion of females (43.0%) in the full data set (n = 5,818). Priority entities, including the side of surgery, ACL graft type, ACL procedure, and meniscal involvement, were identified with F1 scores between 0.87-1. F1 scores of 0.87-1 were attained for several priority entities using a modest set of annotated ACL operative reports to train the model and an NLP tool designed for non-expert use. The NLP pipelines have the potential to extract relevant operative information for a large cohort of patients to support clinical research, such as identifying predictors of subsequent surgery after ACLR. This work can also inform the planning of future studies using larger training sets and more advanced NLP methods.
BACKGROUND
Robotic-assisted total knee arthroplasty (rTKA) is increasingly used because of its surgical precision. However, inconsistent outcomes and high costs often lead patients to seek additional information from artificial intelligence (AI) tools. Large language models (LLMs) such as ChatGPT-4o, Gemini-2.5-Flash, and DeepSeek-V3 are commonly used, but their reliability and readability in orthopaedics remain unclear.
OBJECTIVES
To compare the reliability, usefulness, quality, and readability of responses to common patient questions about rTKA generated by leading LLMs.
METHODS
Three LLMs answered 20 frequently asked patient questions (n = 20) identified through Google Trends and expert validation. Three orthopaedic specialists (n = 3) evaluated reliability, usefulness, and overall quality using validated scales, while readability was assessed with standard indices.
RESULTS
Inter-rater reliability was good to excellent (ICC = 0.728-0.879). Gemini-2.5-Flash achieved significantly higher reliability and usefulness scores than ChatGPT-4o and DeepSeek-V3 (all p < 0.05). ChatGPT-4o and DeepSeek-V3 produced more readable but less accurate content, revealing an inverse relationship between reliability and readability.
CONCLUSIONS
Gemini-2.5-Flash provided the most reliable responses, highlighting the need for supervised integration of LLMs in patient education.
Mehmet Utku Çiftçi, A. Koluman, Ebru Aloğlu Çiftçi et al.· Knee (Oxford)· 0 citations
This review highlights the promise of LLMs in enhancing decision support, workflow efficiency, research productivity, and patient communication in spine care, while emphasizing the need for interdisciplinary collaboration, robust evaluation metrics, and governance frameworks that prioritize patient safety and equity.
Fabio Galbusera, Andrea Cina· European spine journal· 0 citations
This exploratory scenario-based study demonstrates model-specific, rather than universal, language effects on LLM performance in dental trauma management.
PURPOSE
To evaluate current evidence regarding the clinical reliability and reasoning capabilities of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) within the spine surgery workflow. This scoping review utilizes a novel Vertical Integration Maturity Scale (VIMS) to map model maturity across five stages of the vertical workflow, identifying persistent research gaps and technical prerequisites for clinical implementation.
METHODS
A systematic search was conducted across PubMed, Embase, and Scopus for peer-reviewed studies published between January 2023 and January 2026. Utilizing the Population, Concept, and Context (PCC) framework, studies were selected based on their application of generative AI to the clinical evaluation and surgical management of spinal pathologies. Data were analyzed via independent dual-coding using a two-dimensional framework mapping five VIMS levels across five functional workflow stages. Synthesis included categorization by model architecture, input modality, and performance benchmarks, with inter-rater reliability calculated to validate the novel framework.
RESULTS
Of 351 identified records, 40 studies met the inclusion criteria. Research density peaked in Stage III (Evaluative Decision Logic) at VIMS Level 1 (n = 15), indicating a methodological focus on evidence-based guideline retrieval over patient-specific synthesis. While ChatGPT-4 showed high concordance (61.1% to 88.2%) with clinical guidelines, diagnostic precision fell to 11.1% in complex spinal deformity. A critical evidence gap persists in Stage IV (Tactile Procedure Planning) with a total absence of studies evaluating autonomous workflow agency (VIMS 4-5). This absence likely reflects a limitation of retrospective study designs reliant on proprietary, non-version-controlled models. While qualitative synthesis suggests hybrid architectures and Retrieval-Augmented Generation (RAG) may bridge existing semantic gaps, the inferential strength of these trends is currently limited by heterogenous benchmarking metrics.
CONCLUSION
Current LLMs demonstrate proficiency in foundational medical knowledge, but a significant evidence gap remains regarding their inferential validity in the non-deterministic surgical scenarios. Advancing clinical integration requires modular multimodal ensembles that process radiographic data alongside domain-specific reasoning scaffolds. Crucially, future research must transition from retrospective algorithmic benchmarking toward standardized, prospective clinical trials to evaluate algorithmic decision-making against real-world surgical outcomes.
Edwin H. Y. Lui, Ralph J. Mobbs· European spine journal· 0 citations
An open-source, human-verified workflow using large language models can accelerate electronic health record abstraction while improving accuracy and supports broader adoption of transparent artificial intelligence methods in clinical research.
Carl Jannes Neuse, Malte Janssen, S. Ibing et al.· BMC Medical Informatics and...· 0 citations
Objectives: Natural language processing (NLP) can enable scalable extraction of clinically relevant information from unstructured radiology reports retrieved from electronic healthcare data warehouses, but reliance on externally hosted models may pose cost, privacy, and deployment challenges. We compared self-hosted discriminative and generative NLP pipelines for automated extraction of Prostate Imaging and Reporting Data System (PIRADS) scores from multiparametric magnetic resonance imaging (mpMRI) reports used in prostate cancer risk assessment. Materials and Methods: We identified 44,511 mpMRI reports across 68 Veterans Affairs (VA) healthcare systems. A stratified random sample of 1,973 reports was used to train, test, and evaluate multiple pipeline configurations combining Named Entity Recognition (NER) models and large language models (LLMs). Performance was assessed by accuracy of maximum PI-RADS extraction and processing speed using self-hosted implementations of spaCy NER, Transformers NER, and generative LLMs Llama 3, Qwen3, and Gemma3. Results: Across the top 10 pipeline configurations, accuracy for maximum PI-RADS extraction ranged from 89.3% to 95.5%, with processing times spanning 150 milliseconds to 70 seconds per report. Generative LLM pipelines achieved the highest accuracy (up to 95.5%) but were substantially slower (2 to 70 seconds), whereas NER based pipelines demonstrated lower accuracy (88.5%) with faster performance (50 to 150 milliseconds). Discussion: Discriminative NER pipelines achieved high accuracy while offering advantages in speed and potential scalability. Accuracy gains from LLMs were accompanied by significantly higher computational cost, potentially limiting feasibility in high-volume clinical environments. Conclusion: Discriminative methods were more efficient than generative models in annotating PIRADS from mpMRI report text, providing insights into configurations for optimal clinical deployment when volume is a limiting factor. However, generative AI offered improved accuracy with less upfront development.
D. Lee, N. McCoy, C. Haroldsen et al.· medRxiv· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
