MDEF-AI: a multi-level dynamic evaluation framework for assessing the impact of AI-based clinical decision support on decision quality and patient safety

Abstract

Artificial intelligence-based clinical decision support systems (AI-CDSS) are increasingly deployed in healthcare settings, yet their impact on clinical decision quality and patient safety remains poorly understood. The dominant research paradigm evaluates AI-CDSS through algorithmic performance metrics—accuracy, sensitivity, and specificity—that do not capture the cognitive, behavioral, and organizational dynamics through which AI recommendations are translated into clinical decisions. This performance-centric orientation creates a persistent translation gap between demonstrated technical capability and achieved clinical benefit. To propose and theoretically ground a multi-level, dynamic conceptual evaluation framework (Multi-Level Dynamic Evaluation Framework for AI-CDSS; MDEF-AI) that integrates system design characteristics, human–AI interaction mechanisms, contextual moderators, and temporally dynamic feedback loops to provide a comprehensive analytical lens for assessing the impact of AI-CDSS on clinical decision quality and patient safety. A literature review and critical analysis methodology was employed, involving review of 52 peer-reviewed publications across eight thematic domains, thematic analysis, critical evaluation of existing frameworks, and conceptual synthesis. The resulting framework was systematically compared against the three most closely related existing models to establish its distinct theoretical contribution. MDEF-AI is a three-level evaluation structure comprising: (1) contextual moderators across four categories—clinical context, clinician characteristics, organizational factors, and patient characteristics; (2) a core dynamic chain of five interdependent components—system characteristics, human–AI interaction mechanisms (encompassing six mechanisms), clinician cognitive and behavioral responses, clinical decision quality, and patient safety outcomes (encompassing five dimensions); and (3) three temporally distinct feedback loops. AI Omission Trust—the tendency to treat the absence of an AI alert as active clinical reassurance, independent of knowledge of the system’s false negative rate—is proposed as a conceptual interaction failure mode that is hypothesized to operate passively, to resist the design interventions effective against automation bias, and to generate patient safety harm that is largely invisible to existing incident reporting systems. Critical analysis identified six structural gaps in the literature, each corresponding to a specific MDEF-AI design element. Patient safety in the age of clinical AI is an emergent property of the entire sociotechnical ecosystem, not a product of algorithmic performance alone. MDEF-AI provides the theoretical architecture for evaluating AI-CDSS in a way that accounts for this complexity, and generates a structured agenda for empirical validation, instrument development, governance reform, and longitudinal evaluation research.