AI Quality and Clinical Roles Impact Decision Quality in AI-Augmented Medical Decisions More than AI Explanations.
TL;DR
The findings suggest that detailed AI explanations do not necessarily improve AI-augmented medical decisions even though they can increase trust in AI recommendations and the importance of AI recommendation quality in clinical decision making is underscored.
Abstract
Background
Explainable AI has garnered significant attention in recent years, particularly in high-risk domains such as health care. Despite the enthusiasm, empirical evidence on the impact of AI explanations remains mixed. This study explores the effects of AI-generated recommendations and various types of AI explanations on clinical decision quality.
Methods
We conducted 3 experiments involving clinicians who made drug-dosing decisions using AI recommendations of differing quality and explanations. We evaluated 4 types of explanations: a global explanation about how the AI system worked and 3 case-specific local explanations that respectively included the What (what factors were considered for the case), What+Why (why the factors mattered), and What+Why+How (how the system considered them). We recruited clinicians with varying levels of domain expertise, including pharmacists, physicians, nurses, and midlevel providers.
Results
Our results suggest that compared with AI explanations, AI recommendation quality played a more important role in decision quality. Clinical roles, a proxy for domain knowledge, moderated the effects of AI recommendation quality on decision quality and AI influence. Compared with pharmacists and physicians with greater domain knowledge, midlevel providers and nurses were more susceptible to being influenced by low-quality AI recommendations. Contrary to theoretical predictions, local AI explanations increased trust in the AI system but did not improve decision quality over global explanations. Instead, trust in AI seemed largely unrelated to clinical decision quality.
Conclusion
This study underscores the importance of AI recommendation quality in clinical decision making and the need to personalize AI design and AI training to clinicians with different roles and domain knowledge. Our findings also suggest that detailed AI explanations do not necessarily improve AI-augmented medical decisions even though they can increase trust in AI recommendations.
