Performance of large language models in electrocardiogram interpretation: A comparative study.

Comparative Interpretation of Pediatric Electrocardiograms by Large Language Models and a Pediatrician.

A unified 12-lead ECG-language model for interpretation and clinical-endpoint prediction

Automated electrocardiogram (ECG) interpretation has advanced, yet most systems remain narrow classifiers that emit fixed labels rather than the narratives or endpoint-specific answers clinicians need. Generative approaches could instead produce rich narratives, but are constrained by the gap between continuous biosignals and discrete language tokens. Here we present DeepECG-Tok, which reframes ECG interpretation as a unified instruction-following problem. A residual vector-quantization tokenizer (QINCo) maps 12-lead waveforms to language-model-compatible tokens. Its frozen embeddings achieved a macro-averaged area under the receiver operating characteristic curve (AUROC) of 0.96 for 77-condition classification, outperforming supervised and self-supervised baselines. Aligned with a large language model, a single instruction-tuned model performed ECG interpretation, structured reporting and clinical endpoint prediction, including left ventricular ejection fraction, structural heart disease and atrial fibrillation risk, using 7.27 million question answer pairs. Endpoint classifiers using the frozen tokenizer transferred without retraining to four external cohorts, retaining AUROCs of 0.88--0.90. Evaluated end to end using an ontology-grounded large language model as a judge, which achieved a mean agreement (Cohen's K) of 0.82 against two cardiologists, the unified instruction-tuned model scored 0.71 internally and 0.50--0.53 in the same external cohorts. In a blinded reader study, board-certified cardiologists and residents rated its free-text reports comparably to reference clinician reports, with a paired win-tie-loss distribution of 32:35:33 and a forced-choice preference of 0.53 among decided cases, with no significant difference between the model and reference reports in either comparison. These findings establish discrete ECG tokenization as a foundation for general-purpose models that generate clinically useful interpretations and answer diverse questions directly from cardiac waveforms.

Comparative Diagnostic Accuracy of Large Language Models for STEMI Classification

Background: Early and accurate identification of ST-Elevation Myocardial Infarction (STEMI) in the prehospital setting would reduce morbidity and mortality (Rao et al. 2025). Artificial Intelligence (AI) tools may assist emergency medical personnel by providing rapid electrocardiogram (ECG) interpretation and differential diagnosis (Chen et al. 2022; Nallamothu et al., 2015). Objective: To evaluate the diagnostic accuracy of ChatGPT-4.0, Gemini 2.5 Pro, and a hybrid model combining ECG-GPT + ChatGPT-4.0 in classifying STEMI versus non-STEMI cases based on ECG and clinical data inputs. Methods: Fifty-six consecutive de-identified cases (28 STEMI, 28 non-STEMI) from Staten Island University Hospital EMR (1/2025-6/2025) were analyzed. Each case included demographics, vital signs, chief complaints, and a representative 12-lead ECG. The three AI models were independently asked to classify each case as STEMI or not STEMI. Results: For the STEMI cohort (n=28), detection rates were: ChatGPT-4.0 21.4%, Gemini 2.5 Pro 67.9%, and ECG-GPT + ChatGPT hybrid 71.4%. For the non-STEMI cohort (n=28), correct classification rates were: ChatGPT-4.0 92.9%, Gemini 2.5 Pro 28.6%, and ECG-GPT + ChatGPT hybrid: 96.4%. The ECG-GPT + ChatGPT hybrid model demonstrated the highest diagnostic accuracy at 83.9%. Conclusion: While ChatGPT-4.0 showed high specificity, it lacked sensitivity for STEMI detection. Gemini 2.5 Pro improved sensitivity but yielded higher false positives. The hybrid ECG-GPT + ChatGPT model outperformed both standalone models, suggesting that multimodal AI integration may enhance triage accuracy for prehospital care. Despite the small population size and limitations associated with this, these results propose a viable means to hypothesize that AI may have potential in assisting responders with early STEMI identification. Future improvements in model precision could support the implementation of AI-based triage systems in emergency response settings, reducing time to treatment and improving outcomes by aiding direct cardiac catheterization lab transfers without unnecessary delays (Garvey et al., 2012; Squire et al., 2014).

Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis

While multimodal large language models (MLLMs) excel in medical applications, most of them favor static images or short-term signals. In the critical field of dynamic electrocardiograms (ECG), models struggle with complex temporal reasoning and diagnostic report generation due to a lack of high-quality datasets and benchmarks. To address this, we introduce (i) Holtercare-23K, a large-scale multimodal dynamic ECG dataset comprising 22,980 QA pairs derived from 788 clinical Holter records and featuring a novel signal-video-text tri-modal alignment. Based on this dataset, we present (ii) Holtercare-Bench, a multimodal benchmark that evaluates models on temporal localization, clinical diagnosis, and global summarization. Zero-shot evaluations of leading MLLMs reveal a significant performance gap in processing ultra-long pathological sequences. However, fine-tuning representative models yields substantial improvements. This work illuminates the limitations of current MLLMs in electrophysiology and provides a foundational benchmark for long-term medical MLLMs. Our project is available at https://github.com/ZJU4HealthCare/Holtercare-Bench.

ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography

ECGQuest provides a reproducible benchmark for contextual ECG knowledge and shows that parameter-efficient fine-tuning can make smaller language models competitive with substantially larger commercial models.

Beyond accuracy: evaluating the reliability of large language models for medical assessment

For automated assessment metadata extraction, reliability rather than accuracy determines whether an LLM can be deployed, and national origin is not a meaningful predictor.