LLM and CQL computable phenotypes for myocarditis adverse event feature and outcome determinations using patient FHIR data: a feasibility study

Abstract

Post-market surveillance for adverse events of special interest is often constrained by key diagnostic elements being captured primarily in unstructured clinical notes and reports. This feasibility study evaluated large language model (LLM) and clinical quality language (CQL) methodologies to perform phenotype feature extraction and outcome determination on Fast Healthcare Interoperability Resources (FHIR) data. Two health systems identified patients with an mRNA COVID-19 vaccination and an inpatient myocarditis diagnosis within 42 days. Patient FHIR data was retrieved via a Health Information Exchange and converted into narrative text (“textified”). Llama3.3 70B Instruct answered yes/no Brighton Collaborative–based myocarditis feature questions; answers were written back as FHIR Observation resources to enhance each patient bundle. Three methods were compared: simple CQL (structured data only), LLM (textified FHIR data), and enhanced CQL (CQL on enhanced bundles). Three clinicians provided gold-standard determinations. An optimization set ( n  = 10) supported pipeline refinement; performance was evaluated in a test set ( n  = 25). In the test set, simple CQL performed poorly for identifying cardiac and non-specific symptoms [Matthews Correlation Coefficient (MCC): 0.08; F1: 0.02] but excellently for cardiac biomarkers (MCC:0.83; F1: 0.88). The LLM achieved excellent performance identifying symptoms and imaging features (e.g., echocardiogram abnormalities MCC: 0.86) but weaker performance for biomarkers (MCC: 0.54). Enhanced CQL had excellent feature identification performance across most domains. Both LLM and enhanced CQL correctly determined the myocarditis outcome for 23 of 25 patients. In this small feasibility study, structured and unstructured FHIR data provided the computable phenotypes using an LLM with sufficient clinical information for myocarditis feature identification and outcome determination.