Jul 2026· Machine Learning and Knowledge Extraction· Vol 8, pp. 216· 0 citations· 64 references
TL;DR
Overall findings suggest that the effectiveness of decision-support systems depends not only on model performance and explanation quality but also on interaction design, and judicial protocols showed promise in mitigating automation bias and promoting active cognitive engagement in clinical decision support.
Abstract
Artificial intelligence is increasingly used to support clinical decision making, yet concerns remain regarding algorithmic aversion, automation bias and the preservation of meaningful human oversight; while explainable AI aims to improve transparency, less attention has been devoted to the design of human–AI interaction protocols. This study investigates Frictional AI, an interaction paradigm that introduces cognitive friction to encourage critical engagement with AI recommendations. First, semi-structured interviews were conducted with a legal expert and a psychologist and analyzed through thematic analysis to identify legal, ethical, and cognitive requirements for AI-assisted decision support. Second, a user study involving 96 medical residents compared three interaction protocols: a conventional explainable AI-first design (XAI) and two friction-based protocols, namely a judicial protocol based on juxtaposed explanations (Judicial AI, JAI) and an adjunct protocol requiring an initial unsupported decision before AI exposure (AAI). Diagnostic accuracy and confidence, perceived usefulness, completion time, and reliance patterns were evaluated. The interviews highlighted the importance of human-centered explanations, contrastive reasoning, preservation of professional responsibility, and the role of user studies in evaluating human–AI interaction. The quantitative results showed that none of the AI-assisted conditions improved diagnostic accuracy relative to the no-support baseline. However, JAI achieved performance comparable to the baseline, outperforming XAI and AAI, and exhibited the lowest level of over-reliance. Overall findings suggest that the effectiveness of decision-support systems depends not only on model performance and explanation quality but also on interaction design. In conclusion, while preserving diagnostic performance, judicial protocols showed promise in mitigating automation bias and promoting active cognitive engagement in clinical decision support.
Effective mitigation must address both individual barriers, including time pressure, variable AI literacy, and reluctance to challenge automated output, and systemic barriers, including weak governance, opaque tools, misaligned incentives, and inadequate monitoring.
Jeffrey V. Esteron, Rhocette M. Sn Agustin, S. R. Y. Basilio et al.· Patient Education and Counse...· 0 citations
This article addresses the growing concern that generative artificial intelligence (AI) may replace human expertise in organizations. Instead of asking whether AI should be used, it examines why human judgment rooted in experience cannot be fully replaced by current AI systems and how organizations can work with AI more effectively. Drawing on research from cognitive science, neuroscience, and organizational studies, the paper explains how people use prior experience to interpret context, notice subtle cues, and make sense of ambiguous situations—capabilities that differ fundamentally from how large language models process data. Evidence from recent studies of AI use in hiring, performance management, healthcare, and knowledge work shows recurring problems, including mistakes in unusual cases, missed context, over-reliance on AI recommendations, and reduced visibility of real skill differences among employees. In response, we propose a five-part Human–AI Collaboration Framework designed to help organizations use AI for efficiency while keeping human judgment active and accountable in key Human Resource Management decisions. The analysis shows that AI performs best in routine, data-rich situations but falls short when decisions require lived experience and contextual understanding. By framing organizations as systems built on accumulated experience, this article offers practical guidance for responsible AI integration and outlines directions for future research on human–AI collaboration.
Daniel Altieri, Zohra Damani, Cynthia L. Nebel· Administrative Sciences· 0 citations
Many decision support systems (DSS) provide predictions, recommendations, and, increasingly, explanations. Supporting human-AI decision-making with context-specific questions, however, remains largely unexplored. Questions can stimulate reflection and critical thinking, thereby introducing productive friction in the decision-making process and potentially reducing overreliance on DSS. This paper presents a proof-of-concept for generating data-driven questions based on a DSS prediction and its corresponding explanation, i.e., feature contribution, using a local language model. We illustrate our method using a realistic example from the medical field. In informal discussions (n = 2), gathering views on the possible usefulness of questions in decision-making, the clinicians mentioned that the generated questions have potential to help them reconsider the prediction and consider alternative options. Our proof-of-concept informs the design of human-AI interactions aimed at promoting the cognitive engagement of decision-makers and mitigating overreliance on DSS by shifting the focus from explanations to questions.
S. Fischer, Linus Holmberg, S. Thill et al.· Message Understanding Confer...· 0 citations
Abstract Background AI-based clinical decision support systems are increasingly integrated into medical practice, creating hybrid decision-making processes in which physicians and AI systems jointly contribute to clinical judgments. Yet, how different forms of such AI support affect patients’ trust in hybrid medical decisions remains poorly understood. Objective This study aimed to examine how the type and the timing of physician AI support influence potential patients’ trust in the medical decisions, perceptions of the hybrid decision-making process, and intentions to follow the medical advice. Methods In 2 preregistered vignette-based online experiments, 489 (study 1) and 570 (study 2) members of the general public in Germany imagined 4 medical consultations, in which the physician used no AI support, descriptive AI support (informational or visual assistance), or diagnostic AI support (preliminary diagnostic suggestions). Study 2 additionally manipulated the timing of AI support, namely, whether the physician reviewed AI advice after having made an independent own assessment (sequential decision-making) or not (concurrent decision-making). Participants rated their trust in the medical decisions, trustworthiness of the medical provider, uniqueness neglect, and willingness to follow the medical advice on 7-point Likert scales, with greater values representing stronger agreement. Linear mixed-effects models were used for quantitative analyses. Open-ended responses (N=2607) were analyzed qualitatively to identify recurring themes regarding trust in AI-supported decisions. Results In study 1, the physician’s use of diagnostic AI support compared with descriptive AI support produced significantly lower mean ratings of trust in the medical decisions (5.00 vs 5.37; t486=3.51; P=.002) and perceived provider trustworthiness (4.93 vs 5.40; t486=4.46; P<.001), as well as higher mean ratings of perceived uniqueness neglect (3.24 vs 2.93; t486=2.72; P=.02), but no significant differences regarding the willingness to follow the advice (5.42 vs 5.62; t486=1.74; P=.25). Study 2 replicated these results and revealed a significant interaction effect between type and timing of AI support for trust in the medical decisions (t436=2.71; P=.007), perceived provider trustworthiness (t436=2.78; P=.006), and perceived uniqueness neglect (t436=−2.34; P=.02) but not for willingness to follow the advice (t436=1.81; P=.07). Specifically, diagnostic AI support was evaluated less favorably than descriptive AI support when the physician reviewed primary medical information and AI output simultaneously but not when the physician first assessed the primary medical information independently before reviewing the AI output. Qualitative responses showed that participants were concerned that erroneous AI output biases physicians’ judgments and indicated that physician independence could strengthen trust. Conclusions The use of AI to support physicians appears more acceptable when used for analytical rather than decisional support, or when physicians’ decisional independence is visibly retained, suggesting that trust in AI-supported medical decisions depends not only on whether physicians use AI support but also on subjective perceptions of how it is used.
Insa Schaffernak, Julia Cecil, Eesha Kokje et al.· Journal of Medical Internet...· 0 citations
Current evaluation of artificial intelligence (AI) in healthcare remains largely focused on model accuracy, clinical outcomes, efficiency, and the formal availability of human oversight. These dimensions are necessary but insufficient. A system may improve today's decision while, through repeated use, weakening the clinician's ability to recognise tomorrow's error. Cognitive safety is proposed here as a longitudinal property of the clinician-AI-organization sociotechnical system: its capacity to support or improve clinical performance without eroding independent hypothesis generation, uncertainty calibration, reasoned dissent, metacognitive control, and resilient performance when AI is wrong or unavailable. Automation and augmentation should not be treated as ideological alternatives, but as task-sensitive regimes selected according to ambiguity, reversibility, normative content, and the need to preserve skill formation. A Clinical Cognitive Impact Assessment is outlined to make this proposal empirically testable across pre-deployment evaluation and post-implementation monitoring. Keeping a physician formally in the loop is not enough: healthcare systems must preserve over time the cognitive capacities required to understand, challenge, and, when necessary, interrupt that loop.
Salvatore Corrao· Recenti progressi in medicin...· 0 citations
It is suggested in the paper that a methodology of ethical governance based on principles of responsible AI should be structured, fairness-by-design, transparency, human-in-the-loop oversight, and constant impact assessment, which underscores the fact that AI systems have ethical failures that are seldom technical but rather socio-technical, which necessitate interventions at the policy, organizational governance, and technical design levels.
Fatou Diop· International Journal of Inn...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
