Jun 2026· arXiv.org· Vol abs/2606.06081· 0 citations· 44 references
Computer Science
TL;DR
This paper develops the first formal framework for measuring appropriate reliance on set-valued AI advice within the sequential judge-advisor paradigm, spanning both classification and regression tasks.
Abstract
Appropriate reliance on AI advice has become a central research theme in human-AI collaboration. Existing frameworks have focused exclusively on point predictions as AI advice. However, set-valued AI advice (e.g., discrete sets or continuous intervals) is increasingly being used to communicate uncertainty and improve human decision making. In this paper, we develop the first formal framework for measuring appropriate reliance on set-valued AI advice within the sequential judge-advisor paradigm, spanning both classification and regression tasks. For classification, we first introduce the dimensions that are necessary for evaluating set-valued AI advice. We then define two metrics: correct reliance rate on AI and correct reliance rate on self, which jointly characterize appropriate reliance in this setting. For regression, we introduce quantity of AI reliance and quality of AI reliance, which respectively measure whether a decision maker utilized the AI advice and whether their reliance helped them get closer to the ground truth relative to their initial estimate. Through the application of our framework, we demonstrate how these metrics capture important nuances in human-AI collaboration that existing measures overlook.
As generative AI increasingly becomes a common source of daily decision-making, including financial choices, it is critical to understand how people evaluate AI-generated financial advice. We conducted a preregistered vignette experiment (N = 285) in which substantive financial content---including facts, numerical values, recommendation direction, and core reasoning---was held constant while communication style varied across AI Financial Assistant (AI), Certified Financial Planner (Expert), and Online Community Forum (OC) advice. Displayed source attribution was independently manipulated through correctly labeled, unlabeled, and mislabeled conditions, allowing us to separate attribution effects from source-specific communication cues. Expert advice was rated more favorably than AI advice on 9 of 10 outcomes (|d|=0.20--0.47), and this advantage remained visible without source labels, where Expert advice outperformed AI advice on 8 of 10 outcomes (up to d=0.60). Correct labels added limited differentiation, whereas mislabeling increased ratings of AI advice for situational fit and overall quality (d=0.42 for each) and attenuated the Expert advantage in situational fit (d=-0.36). Descriptive analyses further showed that AI advice was most responsive to displayed attribution and, conversely, that advice-style differences were most visible under an AI label. These findings show that financial-advice evaluations are shaped jointly by displayed attribution and message-level communication cues. We position disclosure not as a neutral transparency mechanism, but as an interpretive frame whose accuracy and interaction with message cues can shape trust and reliance.
A. Kapadia, Eshwar Chandrasekharan, Koustuv Saha· 0 citations
As artificial intelligence (AI) systems are increasingly deployed for complex decision-making, calibrating user trust to prevent overreliance on overconfident AI remains a critical challenge. This paper investigates linguistic hedging – the use of tentative language to soften claims and indicate limited certainty – to communicate AI uncertainty, and its impact on user reliance. In a two-part investigation into decision-making in the financial domain, a formative study (N=36) explored strategies for eliciting hedged responses in accordance with model confidence. A confirmatory study (N=71) then measured actual behavioral reliance in a financial investment decision task, manipulating both AI confidence and the decision risk. Our findings reveal that while participants rated hedged and unhedged AI as equally trustworthy and likely to be correct, they were significantly less likely to follow hedged advice in a binary choice. We discuss how linguistic markers can be used to calibrate user reliance to model certainty, reducing overreliance while preserving trust in the system.
Laura Spillner, Johanna Rockstroh, Nina Wenig et al.· International Conference on...· 0 citations
Bayesian Belief Networks (BBNs) are powerful tools for decision-making under uncertainty. However, building their structures and estimating parameters are difficult. Currently, researchers must choose between relying on expert judgement or using large datasets to learn the structure and parameters of the network. We propose a new methodology using Large Language Models to bridge the gap between expert opinion and data-driven learning. This approach uses a panel of AI agents to estimate probabilities based on specific personas and context. We then apply a trimmed-mean rule to remove noise from these responses. We develop a six step BBN framework and illustrate it to model customer intention to consult a doctor in an alternative healthcare system. The model reveals that while self efficacy appears to be a major factor, its actual causal impact is small. In contrast, subjective norms have a much stronger effect in modelling customers'intention. The most effective strategy is to improve both confidence and community norms simultaneously.
Kumar Rahul, Shovan Chowdhury Indian Institute of Management Kozhikode, Kerala et al.· 0 citations
Current AI evaluation frameworks focus primarily on technical performance, including accuracy, robustness, reasoning ability, and policy compliance. These measures remain essential, but they are not sufficient for systems that interact directly with users through natural language. Human-facing AI systems are increasingly used as advisors, coaches, tutors, and companions. In these roles, their responses can shape how users reason, interpret emotions, form beliefs, calibrate trust, and make decisions. The relevant unit of evaluation is therefore not only the model, but the human-AI interaction. This paper introduces psychological competence as a missing dimension in AI evaluation. We define psychological competence as the capacity of a human-facing AI system to support user cognition, emotional interpretation, and behavioral decision-making in ways that are appropriate to the user, context, and purpose of the interaction. This includes interaction properties such as framing, tone, perceived authority, responsiveness, uncertainty handling, and conversational guidance. Existing evaluation approaches capture parts of this problem but rarely assess these psychological effects directly. Drawing on behavioral science and human-AI interaction research, we outline a conceptual framework for psychological competence and its core domains. Rather than proposing a specific benchmark, we define the construct, clarify its boundaries, and describe how it may be assessed through scenario-based probes, structured human evaluation, and model-assisted evaluation methods. We argue that psychological competence should become a core consideration for model providers, deploying organizations, researchers, and regulators concerned with the real-world effects of human-facing AI systems.
M. Economides, Paul M. Sacher, Samuel Salzer et al.· 0 citations
Information-theoretic (IT) measures are ubiquitous in artificial intelligence: entropy drives decision-tree splits and uncertainty quantification, cross-entropy is the default classification loss, mutual information underpins representation learning and feature selection, and transfer entropy reveals directed influence in dynamical systems. Despite wide adoption, measure selection is often decoupled from estimator assumptions, failure modes, and safe inferential claims. This survey provides a practical decision framework for four foundational measures - Entropy, KL divergence/cross-entropy, Mutual Information, and Transfer Entropy - organized around three prescriptive questions for each: (i) what question does the measure answer and in which AI context; (ii) which estimator is appropriate for the data type and dimensionality; and (iii) what is the most dangerous misuse. The framework is operationalized in two complementary artifacts: a measure-selection flowchart and a master decision table. We cover both AI/ML and decision-making agent application domains per measure, with standardized Bridge notes linking IT quantities to cognitive and neuroscientific constructs. Two worked examples illustrate the framework on concrete practitioner scenarios spanning representation learning and temporal influence analysis, and a reproducible multi-agent case study across three learning architectures validates the transfer-entropy surrogate-testing guardrail against a null control.
Nikolaos Al.Papadopoulos, Konstantinos E. Psannis· 0 citations
An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification.
Daniel Romero-Alvarado, Fernando Mart'inez-Plumed, Jos'e Hern'andez-Orallo· Machine-mediated learning· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
MIT News · Artificial Intelligence· news.mit.eduJul 7, 2026
The professor of physics and inaugural director of the NSF AI Institute for Artificial Intelligence and Fundamental Interactions will lead LNS and continue his research in particle physics.
