Beyond Algorithms: Human-Centered Explainability for Clinicians, Patients, and Caregivers
TL;DR
Two short vignettes and a design guide to help human factors researchers create explanation systems that are practical and trustworthy are presented to show how explainability can become part of the care system instead of being treated as a separate technical feature.
Abstract
Artificial intelligence (AI) systems are now used in many areas of healthcare, but stakeholders in the care like clinicians, patients and caregivers still experience mistrust, confusion, and uncertainty when AI-supported recommendations appear in their care. Traditional explainable AI (XAI) methods focus on showing how the algorithm works, which may help developers but often does not support stakeholders in real clinical situations. Human-centered AI research shows that explanations need to match user tasks, mental models, and decision needs. And the SEIPS 2.0 and SEIPS 3.0 frameworks offer a clear structure for placing explanation strategies across the patient and caregiver journey. In this report, we extend two non-algorithmic approaches called Collaborative Explainable AI (CXAI) and Cognitive Tutorials, and we use SEIPS 2.0 and SEIPS 3.0 to guide where and how these methods should be used. We also bring evidence from qualitative studies showing perspective of non-clinical stakeholders on AI in the journey, that patients prefer explanation through conversation, and that caregivers often struggle when AI outputs are unclear or incomplete. We present two short vignettes and a design guide to help human factors researchers create explanation systems that are practical and trustworthy. Our goal is to show how explainability can become part of the care system instead of being treated as a separate technical feature.