Suggestions are provided for the field to transition from individual and post-hoc XAIs toward intrinsically explainable designs where the reasoning logic is built directly into the model architecture to ensure that AI outputs align with human-centric clinical workflows and applications.
Abstract
Multimodal Artificial Intelligence (AI) models-integrating diverse data such as imaging and clinical records-are advancing rapidly in healthcare, yet a significant disconnection persists between these complex predictive architectures and the explainable AI (XAI) techniques used to interpret them. We conducted a scoping review over 4 bibliographic databases to investigate the use of explainability methods in cross-modal medical AI studies. From 82 included studies, we found that the landscape remains dominated by independent feature attribution (assigning importance scores to individual modality in isolation), with the majority of studies relying on post-hoc methods (applied after a model decision is reached) that treat the model as a 'black box'. While emerging trends like visual grounding (linking textual justifications directly to specific image regions) and model reasoning show promise, a critical gap remains in explaining the underlying reasoning process. Standardised evaluation is missing in the majority of studies relying solely on qualitative measures. Only a minority of studies achieve good reproducibility with public codebase. We provide suggestions for the field to transition from individual and post-hoc XAIs toward intrinsically explainable designs where the reasoning logic is built directly into the model architecture to ensure that AI outputs align with human-centric clinical workflows and applications.
This systematic review provides a comprehensive analysis of XAI methods specifically applied to tabular healthcare data for classification tasks, revealing that SHAP remains the dominant post-hoc method, achieving strong model fidelity but showing inconsistent alignment with clinical expert reasoning.
Angelower Santana-Velásquez, M. B. Salazar-Sánchez· Computers· 0 citations
Most artificial intelligence (AI) models used in radiology are black boxes-they produce predictions without explaining the basis of their outputs, raising concerns about clinical safety, accountability, and trust. To address this, a growing body of methods has been developed to help clinicians understand and evaluate AI predictions. This field, known as explainable AI (XAI), aims to help clinicians interrogate, interpret, and critically evaluate AI predictions by identifying factors associated with model outputs. In this educational and practical review, we provide an accessible overview of XAI tailored for practicing radiologists and physicians. We cover the major categories of explanation methods, including saliency maps, perturbation-based and feature-attribution approaches, concept- based methods, and example-based reasoning, as well as uncertainty quantification as a complementary approach for assessing prediction reliability, along with common misconceptions and emerging regulatory obligations. We aim to make XAI easier for healthcare professionals to understand, as effective oversight of AI tools has become a core competency for the modern radiologist.
Gorkem Durak, H. Aktas, Tugba Akinci D’Antonoli et al.· Diagnostic and Interventiona...· 0 citations
It is concluded that explainable artificial intelligence improves trust, reliability, and accountability in healthcare systems and is a prerequisite for successful integration of intelligent technologies into clinical practice.
Riya Jacob K· International Journal of Tec...· 0 citations
A framework through XAI to incorporate various health data sources such as electronic health records, medical imaging, laboratory reports, and wearable sensor information, which can be integrated in the context of achieving higher predictive performance in disease prediction and treatment stratification, as well as decision-making transparency.
M. Aparna, S. Lahane, Dr. Bharti A. Dixit· Journal of Intelligent Decis...· 0 citations
Although promising, LLM-based systems are not yet reliable enough for autonomous medical diagnosis, and multiple recommendations for future research are contained to ensure a high level of safety, transparency, and clinical applicability for LLMs and other AI/ML-related technologies and devices.
M. U. K. Gunawardhna, Pirunthavi Wijikumar, D. Weerasinghe· Sri Lankan Journal of Applie...· 0 citations
Explainable artificial intelligence is widely promoted as the mechanism through which opaque deep learning systems will earn the confidence of clinicians and improve diagnostic performance in medical imaging. The claim has acquired regulatory and commercial momentum faster than it has acquired empirical support, and it has been examined almost entirely in single-modality settings even though contemporary diagnostic models increasingly integrate radiological, pathological, textual and structured clinical data. This review critically evaluates the evidence linking explanation to clinician trust and to diagnostic accuracy, with particular attention to multi-modal imaging contexts. Literature was identified through structured searching of openly accessible scholarly indexes and metadata registries, supplemented by citation tracking and verification of every source against its record of registration. The synthesis distinguishes three largely separate evidence streams that are frequently conflated: technical evaluations of explanation fidelity, human-participant studies of trust and reliance, and normative arguments about transparency. Technical evaluations consistently show that widely deployed attribution maps localise abnormalities less accurately than dedicated detection or segmentation models and than expert annotation, and that several methods are insensitive to the parameters of the models they purport to describe. Human-participant studies are more heterogeneous. Explanations improve accuracy in some reader experiments, particularly for clinicians working outside their task speciality, yet they also increase acceptance of incorrect advice and do not reliably protect against systematically biased models. Effects depend on explanation format, reader expertise, the correctness of the underlying prediction and the way trust is operationalised, which varies markedly across studies. Evidence specific to multi-modal systems is scarce, attribution methods rarely apportion importance across modalities in a clinically meaningful way, and evaluation frameworks derived from single-image tasks transfer poorly. The most defensible conclusion is that explanation is neither a general solution to opacity nor an irrelevance, but a design variable whose effect on reliance is conditional and occasionally harmful.
Ogechi Okoroafor, M. S. Ibrahim, Chimuanya Ibecheozor et al.· Journal of Advances in Medic...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
