PharyTriFuse: Knowledge- and LLM-Augmented Deep Learning for Bacterial Pharyngitis Detection from Smartphone Throat Images

Abstract

Bacterial pharyngitis requires timely antibiotic treatment, whereas most non-bacterial cases are self-limited; diagnostic errors may therefore lead to missed infections or unnecessary antibiotic use. This study proposes PharyTriFuse, a multimodal framework that integrates throat-image analysis with large language model (LLM) reasoning and a medical knowledge graph (KG) to classify bacterial versus non-bacterial pharyngitis from smartphone-acquired oropharyngeal images. Experiments were conducted on the public PGUPharyngitis dataset over 742 images using a stratified 72%/8%/20% train/validation/test split. Images were standardized using CLAHE and redness enhancement to reduce acquisition variability. Two visual backbones (EfficientNet-B4 and ConvNeXt-Base) were evaluated under four configurations: AI-only, AI+LLM, AI+KG, and AI+LLM+KG. Performance was assessed using standard classification metrics and inference efficiency. Results show that incorporating LLM reasoning and structured medical knowledge improves classification performance over vision-only baselines while maintaining real-time inference capability under certain configurations. These findings suggest that multimodal AI systems can enhance smartphone-based decision support for pharyngitis assessment.