Skip to content
Review Open access

Trust by design in AI-augmented procurement systems: the roles of explainability, governance, and human oversight

Jul 2026 · Frontiers in Artificial Intelligence · Vol 9 · 0 citations · 37 references
Medicine

TL;DR

It is found that AI use is associated with higher perceived negotiation efficiency, but also with lower AI system trust when automation displaces relational cues and explanations are weak, and multi-group analyses indicate stronger XAI–trust associations in the EU and stronger HIL associations in ASEAN.

Abstract

With the growing integration of artificial intelligence (AI) into buyer–supplier negotiations, procurement teams must translate efficiency gains into defensible and appropriately calibrated reliance on AI-mediated decision support. This study develops and empirically evaluates a socio-technical trust-by-design model for AI-augmented procurement negotiation systems. It jointly considers perceived transparency/explainability (XAI), ethical governance visibility, and human-in-the-loop (HIL) relational design as conceptually distinct levers of AI system trust—that is, professionals’ willingness to rely on an AI negotiation system and their belief that it behaves competently and responsibly. AI system trust is treated as a focal proximal outcome, not as a proxy for, or a sufficient explanation of, dyadic buyer–supplier trust. Using an exploratory sequential mixed-methods approach—six multi-sector case studies across the EU and ASEAN regions (36 interviews) followed by a purposively recruited cross-sectional survey of 238 AI-exposed professionals analyzed via structural equation modeling (SEM)—we find that AI use is associated with higher perceived negotiation efficiency, but also with lower AI system trust when automation displaces relational cues and explanations are weak. XAI exhibits the strongest positive association with AI system trust, while HIL design and governance show additional positive associations. In an augmented specification, XAI and HIL statistically account for the AI use–trust association. A small inverse-U pattern further suggests that moderate, well-governed AI use is associated with higher trust than very low or very high automation intensity. Multi-group analyses indicate stronger XAI–trust associations in the EU and stronger HIL associations in ASEAN. These findings contribute to trust-in-AI and procurement research by (i) clarifying the boundary between trust in an AI system and trust in a buyer or supplier, (ii) specifying procurement-specific conditions—confidentiality, auditability, and negotiation tacticity—under which AI use can erode confidence in AI support, and (iii) offering a cautiously framed, evidence-consistent roadmap for trustworthy AI negotiation systems. The reported associations should be interpreted within a non-probability sample and do not establish population-representative effects or buyer–supplier relationship outcomes.

Read PDF

Similar papers

Open access 2026

Calibrated Trust in Agentic AI Work Systems: A Social Trust Calibration Framework for Responsible Human-AI Adoption

Agentic artificial intelligence (AI) is transforming social and organisational life by moving AI from an assistive tool into a semi-autonomous actor that can plan, recommend, communicate, coordinate workflows and trigger decisions. Existing debates on trustworthy AI often emphasise technical reliability, legal compliance or ethical principles, but social adoption depends on a subtler question: how do people learn when to trust, distrust, verify or refuse AI outputs in everyday work? This conceptual research develops the Social Trust Calibration Framework (STCF) for responsible human-AI adoption in organisations and public institutions. Using an integrative conceptual synthesis of trust theory, automation research, organisational sensemaking, algorithmic management, human-centred AI and contemporary AI governance guidance, the study identifies the mechanisms through which AI trust becomes either calibrated, excessive, deficient or displaced. The resulting framework argues that trust in agentic AI must be calibrated across four mutually dependent layers: capability trust, process trust, institutional trust and identity trust. The paper contributes a formal definition of calibrated AI trust, a social calibration loop, a diagnostic matrix of trust states, a maturity model and seven research propositions for future empirical testing. The analysis shows that appropriate reliance cannot be achieved by accuracy alone. It requires visible evidence, role-specific verification routines, accountable decision rights, psychological safety, transparent escalation paths and protection of human agency. The paper concludes that responsible AI adoption should be treated as a social trust calibration problem rather than a simple technology acceptance problem. This reframing offers practical guidance for leaders, educators and policymakers seeking to scale AI while preserving human judgement, legitimacy and social confidence.

K. Tan · 0 citations
Review Open access Jul 2026

Human-AI Collaboration and Trust in Human Resource Management: Opportunities, Challenges and Framework for Fair Implementation

AI is being increasingly integrated into HRM, modifying recruitment, performance analysis, learning and development, and managerial decisions. This paper uses eighteen recent organizational psychology studies and focuses on information systems and human-computer interaction. It provides a synthesis on what triggers trust between employees and AI, the human-algorithm workload balance, and fairness concerns regarding AI in people management and assessment. The review concludes that trust in AI is separate from trust in human colleagues, that the process shapes trust and the delegation of trust can improve employees’ performance and overall satisfaction, even when the trust is not complete. This review also notes that trust, explainability, accountability, and the varying perceptions of fairness limit the deployment of AI. Task Division, trust construction, varying perceptions of fairness, and the organizational structure are the main components of the Human-AI and HRM collaboration system, which this review seeks to examine. It also identifies research gaps.

Nidhi Goel · 0 citations
Review Open access Jul 2026

From institutional trust to AI adoption: a trust transfer and risk perception model of AI-enabled public service acceptance in China's digital government context

Introduction Artificial intelligence (AI) is increasingly used in public agencies to route inquiries, screen eligibility, support caseworkers, and automate routine service encounters. Citizen acceptance of these services depends on their links to public authority, accountability, and visible opportunities for human recourse. This study examines a trust-based mechanism connecting institutional trust, risk perception, AI service trust, and behavioral intention in China's digital government context. Methods The study combined an LLM-driven agent simulation involving 936 agents across three independent seeds, a 3 × 3 factorial scenario experiment involving 900 simulated agents, and a human-validation pilot using the same questionnaire and scenario structure. The pilot generated 189 submitted records, of which 182 were retained after attention checking. Results In the synthetic calibration, institutional trust is positively associated with AI service trust (IT → AST β = 0.607) and negatively associated with risk perception (IT → RP β = −0.271); risk perception is negatively associated with AI service trust (RP → AST β = −0.459); and AI service trust is positively associated with behavioral intention (AST → BI β = 0.424). The same directional pattern appears in the human-validation pilot (IT → AST β = 0.357; IT → RP β = −0.240; RP → AST β = −0.513; AST → BI β = 0.650). Scenario means also align with the simulation pattern (Pearson r = 0.803 for AST and r = 0.875 for BI across the nine cells), with the lowest pilot AST (3.667) and BI (3.413) in the fully automated high-risk condition. Discussion The findings connect confidence in government institutions with service-specific trust and indicate that perceived risk constrains acceptance of AI-enabled public services. In high-stakes automated settings, visible arrangements for human review may be necessary for AI service trust to translate into intended use. Public-sector AI acceptance is therefore shaped jointly by institutional credibility, perceived risk, and service encounter design.

Huihui Wang, Shixin Zhu · 0 citations
Review Open access 2026

Trust in the machine: Expanding UTAUT with competence trust and user experience in organizational AI platform

As more and more AI-powered tools and platforms are adopted in organizations to automate routine tasks, support decision-making, and improve efficiency, adoption is often lopsided, with employees embracing the system while also wondering whether it can effectively perform work-critical tasks. This study examines how organizations adopt AI platforms by applying the Unified Theory of Acceptance and Use of Technology (UTAUT) in the context of user experience (UX) and competence trust, two AI-salient concepts. Competence trust is the extent to which employees believe an AI platform will reliably produce accurate, dependable, and work-relevant outputs. An exploratory sequential mixed-methods design was employed to generate and confirm inductive analysis-level explanations of trust formation and acceptance, grounded in insights gained through observation. The first phase involves semi-structured interviews with 15–25 organizational users. In this phase, the study maps the path from the UX stage to trust or distrust and acceptance, identifies important incidents that affect people's confidence in the platform, and gathers users’ trust-related language to help fine-tune constructs and measurement criteria. In Phase 2, a survey instrument is developed from issues identified in Phase 1 and established scales. CFA and SEM test a longer UTAUT model in which factors affecting performance expectancy, effort expectancy, social influence, and facilitating conditions are used to quantify levels of competence, trust and behavioral intention to use the AI platform. Where the sample size allows, multi-group comparisons can be made by user intensity or job function. Phase 3 combines qualitative themes and quantitative path findings through a combined display to extract converging data, identify contradictions, and provide expanded interpretations, ultimately deriving actionable suggestions that will be applied. Contribution of the study. The study contributes theoretically by framing competence trust as an integral mechanism linking UX to adoption within a broader UTAUT framework, and by providing a better explanation of why perceiving usefulness and ease of use alone may be inadequate in AI settings. Moreover, it has practical design and governance implications for organizations to foster sustainable adoption by incorporating UX features that signal reliability (e.g., stability, clear guidance, robust error handling) and by reinforcing social and organizational support to enhance trust.

Unknown authors · 0 citations
2026

Trust Under Techno-Pressure: A Mixed-Methods Study of Human–Artificial Intelligence Collaboration in the Garment Factories

The rapid adoption of artificial intelligence (AI) in labor-intensive manufacturing raises concerns about how trust between humans and AI develops under production pressure. This study examines the erosion and consequences of human–AI trust in garment factories, where workers must quickly adapt to AI-driven systems in highly monitored environments. Drawing on the Swift Trust Theory and the Job Demands–Resources model, we propose a framework that considers relationships among constructs, such as compressed trust formation, trust fragility, sacrificial compliance, perceived organizational support, and workplace techno-pressure. We employed a two-phase mixed-methods design. An exploratory qualitative study informed construct development, followed by a quantitative study for scale validation and hypothesis testing. Results show that compressed trust formation is positively associated with trust fragility, and both are positively linked to sacrificial compliance. Trust fragility partially mediates the relationship between compressed trust formation and sacrificial compliance. Perceived organizational support weakens the relationship between compressed trust formation and trust fragility, whereas workplace techno-pressure strengthens the relationship between trust fragility and sacrificial compliance. The findings suggest that trust formed rapidly under techno-pressure can enable short-term coordination but remains structurally fragile and may convert into self-sacrificial work behaviors. The study extends Swift Trust Theory to human–AI collaboration and embeds trust dynamics within the Job Demands–Resources model, highlighting how organizational support and techno-pressure management shape whether digital transformation supports sustainable or harmful forms of adaptation.

Surajit Bag, Muhammad Sabbir Rahman, S. Alam · 0 citations
Jul 2026

Rethinking managerial rationality in the age of AI: a human–machine collaboration perspective on organizational decision-making

This paper reconceptualizes managerial rationality in artificial intelligence (AI)-augmented decision-making through the notion of algorithmic-bounded rationality (ABR). It argues that AI does not remove boundedness but relocates it into algorithmic constraints related to data volatility, model opacity and governance maturity. Building on bounded rationality and socio-technical systems theory, this conceptual study develops an ABR framework linking three decision modes (AI-led, human-first and collaborative) to mechanisms of algorithmic boundedness. The framework is further extended through propositions on mode–task fit and governance conditions for sustaining hybrid decision architectures. The analysis shows that human–AI collaboration represents a distinct rationality configuration rather than a midpoint between automation and human judgment. Under ABR, each decision mode becomes effective under different combinations of data intensity, contextual ambiguity and accountability demands. Managers should treat AI integration as a redesign of decision governance rather than a technological upgrade, emphasizing appropriate authority allocation and oversight mechanisms. The study reframes rationality in the AI era by showing how boundedness shifts from human cognition to socio-technical decision infrastructures. It contributes a mechanism-based framework linking decision modes, task conditions and governance arrangements in AI-augmented decision systems.

Z.-S. Chen · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.