T-MoXAI (Temporal Multimodal eXplainable AI) provides a baseline for temporal multimodal XAI, addressing fragmentation in the field and supporting applications where understanding decisions is as important as predictive accuracy.
Abstract
Artificial Intelligence (AI) models for temporal multimodal data have potential in healthcare and agriculture, but their opacity can limit trust and adoption. We introduce T-MoXAI (Temporal Multimodal eXplainable AI), a hierarchical framework explaining (1) when timepoints influence predictions, using temporal Shapley values; (2) which modalities contribute at those moments, using attention analysis; and (3) what features or image regions drive decisions, using gradient based attribution. A transformer based architecture handles irregular temporal sequences and heterogeneous data, generating all three explanation levels in under one second for interactive decision support. We evaluate the framework on two real world tasks: predicting IVF treatment outcomes from ultrasound sequences and clinical measurements (AUC 0.660 despite significant class imbalance), and forecasting wheat yield from temporal RGB imagery and phenotypic traits ($R^2$ 0.265 amid substantial environmental variability). Ablation studies indicate that temporal modelling is critical in both domains: removing it reduces performance to the equivalent of random guessing. Temporal ROAR experiments provide evidence that the explanations reflect the model's reasoning process. With a unified, domain agnostic architecture and open source implementation, T-MoXAI provides a baseline for temporal multimodal XAI, addressing fragmentation in the field and supporting applications where understanding decisions is as important as predictive accuracy.
A unified learning paradigm is proposed that jointly optimizes predictive performance and modality-specific explainability by identifying the most informative elements inside each modality that contribute to diagnostic decisions by employing a matrix-based Renyi's $\alpha$-order entropy functional under the assumption...
Yu-Qing Yang, Alexander Schmatz, Zhao-Zhao Ma et al.· 0 citations
The Integrated Multi-Modal Contextual Network (IMCN) is proposed, a predictive modelling framework that combines multi-modal transformer networks for cross-modal fusion, recurrent neural networks with attention for real-time sequential signals, pre-trained autoencoder networks for dimensionality reduction of high-dimen...
V. R, Madala Guru Brahmam, Alagiri I· Scientific Reports· 0 citations
This work presents the basic principles, techniques, and evaluation frameworks for developing explainable multimodal AI systems that effectively balance accuracy and transparency with reliability and investigates the impact of explainability on reliability, fairness, and user trust in high-stakes sectors including heal...
B. Rajalingam, D. Soundaravalli, S. Kowsalya et al.· ITM Web of Conferences· 0 citations
Heart disease remains one of the primary causes of mortality worldwide, creating a critical need for accurate and interpretable diagnostic systems that support timely clinical intervention. Although numerous machine learning and deep learning approaches have been developed for heart disease prediction, many existing mo...
Jeyadevan S, C. Amuthadevi, G. Rajalakshmi et al.· 2026 2nd International Confe...· 0 citations
MetaCAM is discussed, a novel XAI framework that integrates multiple techniques to produce more stable, robust and clinically meaningful visual explanations and underscores that future progress in medical and sports‑medicine AI depends not only on predictive accuracy but also on transparent reasoning that clinicians, p...
A. Kruglikov, K. Dick, Heather E. Howley et al.· Health Science Inquiry· 0 citations
Experimental results show that the proposed GuardMLLM improves performance on tasks such as predicting patient mortality and ICU length of stay, and effectively alleviates overconfidence in LLM.
Gang Fu, Xiao-Long Xu, Hao-Long Xiang et al.· IEEE journal of biomedical a...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026