Graph-Aware and Sequence-Aware Multimodal Deep Learning Framework for Cancer Detection and Risk Analysis from Medical Imaging
TL;DR
A graph-aware and sequence-aware deep learning framework that combines a convolutional neural network backbone for spatial feature extraction, a Graph Attention Network for lesion-structure modelling, and a Bidirectional Long Short-Term Memory (BiLSTM) module for ordered-view or slice-sequence representation learning is proposed.
Abstract
Early and accurate cancer detection from medical imaging remains challenging because clinically relevant evidence is distributed across local image appearance, structural relationships between suspicious regions, and ordered imaging context. This study proposes a graph-aware and sequence-aware deep learning framework that combines a convolutional neural network (CNN) backbone for spatial feature extraction, a Graph Attention Network (GAT) for lesion-structure modelling, and a Bidirectional Long Short-Term Memory (BiLSTM) module for ordered-view or slice-sequence representation learning. Cross-attention-based multimodal fusion is evaluated exclusively for the RSNA mammography task, where the metadata branch is restricted to patient age and implant status, both available before diagnosis. In contrast, the primary LIDC-IDRI experiment is conducted as an image-only analysis because radiologist malignancy scores and semantic nodule attributes are annotation-derived variables and are not treated as independent clinical predictors. The framework is evaluated on the RSNA Breast Cancer Detection dataset and the LIDC-IDRI lung CT dataset using accuracy, precision, recall, F1-score, specificity, and area under the receiver operating characteristic curve (AUC). Additional ablation experiments assess the contribution of graph learning, sequence-aware modelling, and leakage-safe RSNA metadata fusion, while SHAP analysis quantifies the influence of the included RSNA metadata variables on multimodal predictions. For the RSNA multimodal experiment, the best held-out run achieved 95.2% accuracy and an AUC of 0.978, while three repeated runs yielded 94.0 ± 0.3% accuracy and an AUC of 0.970 ± 0.007 under the internal patient-wise benchmark protocol. These results should be interpreted as public-dataset benchmark outcomes rather than evidence of real-world clinical performance; external multi-centre and prospective validation is required before clinical deployment.
