Nov 2026· Proceedings of the 34th ACM International Conference on Multimedia· pp. 12192-12201· 0 citations· 41 references
TL;DR
CGFusion, an interpretable multimodal framework that jointly models representation alignment and concept-level semantic grounding, is proposed, suggesting that explicitly coupling multimodal alignment with concept-level grounding is a promising direction for building reliable and interpretable multimodal medical systems.
Abstract
Integrating radiology, whole-slide pathology, and structured clinical data is essential for reliable cancer prediction, yet remains challenging due to substantial cross-modal heterogeneity. Existing multimodal models often fail to establish consistent cross-modal representations and provide limited semantic interpretability, making their predictions difficult to trust in clinical settings. To address these challenges, we propose CGFusion, an interpretable multimodal framework that jointly models representation alignment and concept-level semantic grounding. First, a patient-centric contrastive module explicitly aligns CT and WSI representations at the patient level, reducing semantic inconsistency between macroscopic radiology and microscopic pathology. Second, structured clinical variables are projected to initialize the latent queries of a Perceiver-based fusion network, enabling patient-specific priors to guide cross-modal interaction from the outset. Third, an LLM-guided concept grounding module anchors fused latent tokens to clinically meaningful prototypes through temperature-scaled similarity, token-dependent abstention, and hierarchical regularization. Experiments on NLST and MMIST ccRCC demonstrate that CGFusion achieves strong overall performance on NLST and delivers more balanced predictions than competitive baselines on MMIST under severe class imbalance. These results suggest that explicitly coupling multimodal alignment with concept-level grounding is a promising direction for building reliable and interpretable multimodal medical systems.
PURPOSE
Pretreatment estimation of pathological complete response (pCR) to neoadjuvant chemotherapy may support risk stratification and future response-adapted clinical studies in breast cancer. However, existing deep learning approaches often rely on "black-box" visual pattern recognition, suffering from poor interpre...
The integrative analysis of histopathological Whole-Slide Images (WSIs) and transcriptomic profiles holds significant promise for cancer survival prediction. However, existing methods typically project multi-modal features directly into a shared latent space without explicit alignment, leading to the entanglement of mi...
Tao Zhou, Ying Hu, Hua-Zhu Fu et al.· IEEE Transactions on Pattern...· 0 citations
CRUX (CRoss-modal Unification via teXt anchoring), a fusion framework that uses clinical text---encoded by a biomedical language model---as an institution-invariant semantic anchor for robust, generalizable multimodal cancer prognosis, is presented.
Accurate prognosis prediction for head and neck cancer typically necessitates integrating highly heterogeneous clinical and pathological evidence by sophisticated multimodal fusion. However, existing fusion techniques are usually grounded in case-generic clinical rationales about complementary synergies across modaliti...
Yue-Zhao Lin, Shu-Chang Ye, Sheng-Hai Wu et al.· International Conference on...· 0 citations
Integrating heterogeneous cancer data, such as histopathological images and genomic profiles, provides a multimodal perspective for assessing patient survival and improves predictive performance. However, existing methods often overlook the complex entanglement among multimodal shared prognostic information (consistenc...
Geng Gao, Kai-Xuan Zhang, Ni-Ni Rao· Proceedings of the 34th ACM...· 0 citations
Clinical diagnosis is inherently a structured reasoning process, yet existing deep learning models often bypass this structure by mapping image features directly to disease labels without explicitly interrogating the morphological and textural criteria that clinicians systematically evaluate. This limits diagnostic tra...
Asim Khan, Samee Ullah Khan, D. Mahapatra· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.