Pilot comparison of nine large language models generating radiological reports from dental panoramic radiograph pathology notation

Abstract

Abstract Large language models (LLMs) may help convert structured pathology notations derived from dental panoramic radiographs (DPRs) into radiological reports, but comparative data remain limited. This pilot head-to-head study evaluated nine publicly available LLMs across five domains: correct tooth numbering, correct diagnosis assignment, avoidance of confabulations, proper radiological terminology, and avoidance of repetitions. Nineteen anonymized DPRs from an open-source repository were converted into standardized pathology notations by an experienced clinician. Nine LLMs generated reports from identical prompts and identical input data without task-specific fine-tuning. Report quality was rated on a 3-point ordinal scale. Between-model differences were assessed using the Friedman test with Nemenyi post-hoc comparisons; accuracy and F1-scores were also calculated. Significant between-model differences were found in correct diagnosis assignment, avoidance of confabulations, and proper radiological terminology, whereas differences were smaller in correct tooth numbering and avoidance of repetitions. The strongest overall performance was achieved by o1 and o3-mini (accuracy: 0.91 and 0.92; F1-score: 0.95 and 0.96), with o3-mini-high also performing strongly (accuracy: 0.87; F1-score: 0.93). No model consistently met expert expectations across all domains. Performance differed significantly between models. Reasoning-oriented models, particularly o1 and o3-mini, performed best overall, especially in correct diagnosis assignment and avoidance of confabulations. LLM-assisted radiological reporting appears promising, but current performance does not support autonomous clinical use and still requires expert oversight. Moreover, the findings reflect model versions available at the time of testing and should not be interpreted as a definitive ranking of LLM performance.