How Well Do AI Chatbots Understand Abnormal Anatomy: A Comparative Study Using Congenital Anomalies and Tumor Cases

Abstract

Artificial Intelligence (AI) chatbots are becoming an efficient option to understand medical data and also help with clinical reasoning. There has been a recent progression in research of large language models and their ability to be used in the healthcare sector, such as radiological image analysis, and diagnostic support. There is however, little evidence supporting their ability to accurately understand abnormal anatomical conditions, such as congenital anomalies and tumor related changes in anatomy. To assess the diagnostic capabilities of three popular AI-powered chatbots (ChatGPT, Gemini, and Microsoft Copilot) in the interpretation of representative cases of radiological congenital anomalies and tumors in these. A comparative cross-sectional study was done on 20 representative radiological cases which comprised 10 congenital anomalies and 10 tumor cases. Each AI chatbot was shown a set of standard questions and images, respectively. Evaluations of the responses were based on the diagnostic accuracy, anatomical description, embryological or pathological explanation, and the completeness of the interpretation. The analysis of the overall performance was based on the number of correct answers and accuracy of responses. Among the three, ChatGPT had the most accuracy with 19 correct responses (95.0%), followed by Gemini with 17 correct (85.0%) and Microsoft Copilot with 16 correct (80.0%). ChatGPT always gave the most detailed explanation of the anatomical and pathological processes. Microsoft Copilot correctly identified the diagnosis in most cases, with a high rate of correct answers and only limited confusion with similar congenital anomalies, while Gemini was more successful at making the diagnosis but sometimes mixed it up with similar congenital anomalies and less detailed with anatomical description, especially in more complex cases of tumor. All three AI chatbots exhibited a high level of ability in the understanding of radiological cases with abnormal anatomy, indicating their potential use as a supplementary tool in radiological assessment and radiological diagnostic support. Although diagnostic accuracy and the degree of anatomical interpretation varies, however, there are a number of ways in which the responses generated by AI should be interpreted with caution and checked by qualified healthcare professionals before it could be used in clinical settings.