Comparative analysis of discriminative and generative natural language processing pipelines for automated prostate magnetic resonance imaging reports
TL;DR
Self-hosted discriminative and generative NLP pipelines were more efficient than generative models in annotating PIRADS from mpMRI report text, providing insights into configurations for optimal clinical deployment when volume is a limiting factor.
Abstract
Objectives: Natural language processing (NLP) can enable scalable extraction of clinically relevant information from unstructured radiology reports retrieved from electronic healthcare data warehouses, but reliance on externally hosted models may pose cost, privacy, and deployment challenges. We compared self-hosted discriminative and generative NLP pipelines for automated extraction of Prostate Imaging and Reporting Data System (PIRADS) scores from multiparametric magnetic resonance imaging (mpMRI) reports used in prostate cancer risk assessment. Materials and Methods: We identified 44,511 mpMRI reports across 68 Veterans Affairs (VA) healthcare systems. A stratified random sample of 1,973 reports was used to train, test, and evaluate multiple pipeline configurations combining Named Entity Recognition (NER) models and large language models (LLMs). Performance was assessed by accuracy of maximum PI-RADS extraction and processing speed using self-hosted implementations of spaCy NER, Transformers NER, and generative LLMs Llama 3, Qwen3, and Gemma3. Results: Across the top 10 pipeline configurations, accuracy for maximum PI-RADS extraction ranged from 89.3% to 95.5%, with processing times spanning 150 milliseconds to 70 seconds per report. Generative LLM pipelines achieved the highest accuracy (up to 95.5%) but were substantially slower (2 to 70 seconds), whereas NER based pipelines demonstrated lower accuracy (88.5%) with faster performance (50 to 150 milliseconds). Discussion: Discriminative NER pipelines achieved high accuracy while offering advantages in speed and potential scalability. Accuracy gains from LLMs were accompanied by significantly higher computational cost, potentially limiting feasibility in high-volume clinical environments. Conclusion: Discriminative methods were more efficient than generative models in annotating PIRADS from mpMRI report text, providing insights into configurations for optimal clinical deployment when volume is a limiting factor. However, generative AI offered improved accuracy with less upfront development.