MAMGAT-Net: A Multimodal Graph Attention and Multi-Scale Feature Fusion Framework for Early Breast Cancer Diagnosis from Mammography and Histopathological Images

Abstract

Breast cancer is one of the major causes of death among women worldwide, and timely and accurate diagnosis has proven to play a critical role in increasing breast cancer patient survival. Traditional single-modality diagnostic systems, however, tend to be unable to capture both the macro-level structural abnormalities that can be seen in mammograms and the micro-level cellular characteristics seen in histopathological images. To overcome this, this research introduces a novel framework called MAMGAT-Net for the diagnosis of breast cancer, which combines the multimodal approach with Graph Attention and Multi-Scale Feature Fusion networks. To address this challenge, this work proposes a novel framework, MAMGAT-Net, which integrates the multimodal approach, Graph Attention, and Multi-Scale Feature Fusion networks for breast cancer diagnosis. The proposed architecture combines the multi-scale convolutional mammography branch and ResNet-based histopathology branch with a dual-stream feature extraction mechanism. A cross-modal attention mechanism is used to learn to align and fuse complementary diagnostic information across modalities, in a bidirectional fashion. Then, a multimodal diagnostic region graph is built with a Cross-Gated Multi-Head Graph Attention Network to capture complex relational relationships among multimodal diagnostic regions, and global graph pooling and feature refinement are used to provide robust classification. Experimental results on CBIS-DDSM and BreaKHis datasets show that MAMGAT-Net can outperform the conventional machine learning and deep learning baselines with 94.59% accuracy, 92.90% precision, 96.56% recall, 94.70% F1-score, and 98.58% AUC. The effectiveness of multi-scale feature extraction, attention-based fusion, and graph relational learning is further validated in ablation studies. Furthermore, the proposed framework is interpretable and clinically relevant based on the Grad-CAM and Integrated Gradients analysis. The results show that the multimodal breast cancer diagnosis solution of MAMGAT-Net is effective and reliable.