A comparative study of large language models in responding to breast cancer–related questions

Abstract

Breast cancer is one of the most common cancers among women worldwide. With advances in diagnostic and therapeutic technologies, more patients seek information about their condition. Large language models (LLMs) have attracted attention in healthcare for their ability to produce personalized health content. This study aims to assess the performance of diverse LLMs in answering breast cancer-related questions under different contexts. This study will evaluate the performance of several LLMs in providing health information support for breast cancer, and the assessment criteria include medical accuracy, completeness, helpfulness, safety, readability and linguistic complexity of the generated content. Five LLMs were evaluated on breast cancer multiple-choice questions, clinical case analyses, and patient concerns. Three specialists blindly rated responses for completeness, correctness, readability, helpfulness, and safety. Text readability was analyzed using LDU-TGP. Statistical analyses included Cochran’s Q, McNemar’s test, Intraclass correlation coefficient (ICC), linear mixed-effects models (LMM), and Bonferroni correction applied for multiple comparisons. In multiple-choice tests, LLM accuracy rates ranged from 82.22% to 94.44%, with no significant pairwise differences. LDU-TGP platform found that LLM-generated texts had varying linguistic complexities, with ChatGPT-5.2 having a higher reading difficulty score and DeepSeek having a lower and more stable score. Three breast cancer experts independently assessed the sense of concern for these patients 1,500 times and obtained a total mean score of 3.982 ± 0.460. LMM model analysis showed that the LLMs performed differently in the evaluation. ChatGPT-5.2 and DeepSeek had relatively high completeness scores (4.350 ± 0.075 and 4.275 ± 0.075, respectively), but Gemini 3.0 achieved the highest readability score (4.108 ± 0.075). LLMs performed similarly on standardized breast cancer knowledge .LLMs showed similar results on the standard breast cancer knowledge test but had different linguistic features and multi-dimensional response quality. No single model outperformed the others in all the test areas, so many different LLMs were chosen to provide support for breast cancer health information.