Enhancing medical Q&A systems with multimodal knowledge graphs and dual-layer attention mechanisms

Abstract

Medical intelligent question-answering (QA) systems have become important tools for improving the efficiency of healthcare services, and recent research has increasingly emphasized performance optimization and multimodal integration. However, existing systems still face several challenges in intent recognition, entity extraction, and multimodal knowledge fusion, particularly reduced accuracy in multi-label classification, heavy reliance on large-scale annotated data, and limited support for cross-modal retrieval. To address these issues, this study proposes a medical intelligent QA framework that integrates a dual-layer attention mechanism, a large language model, and a multimodal medical knowledge graph to improve system understanding and response generation in complex clinical scenarios. Specifically, we develop a text-based intent recognition model with a dual-layer attention architecture, in which a global contextual attention module is introduced to capture long-range semantic dependencies and improve multi-label classification performance. In addition, an instruction-tuned large language model is employed for zero-shot medical entity recognition, thereby reducing dependence on manually annotated datasets. Building on this foundation, we construct a multimodal medical knowledge graph comprising more than 15,000 associated medical images and develop a visualization-oriented retrieval interface using Flask and ECharts. Experimental results show that the proposed intent recognition model achieves a peak Micro-F1 of 94.42% on multiple benchmark datasets, outperforming several baseline methods. The LLM-based entity recognition module achieved competitive recall in medical entity extraction, demonstrating strong capability in identifying medical entities. User evaluation results further indicate that the system is effective and practical across a variety of medical query types. This study provides a feasible framework for advancing medical QA systems through improved intent recognition, low-resource entity extraction, and multimodal knowledge integration.