Skip to content

Simulated Morality, Misplaced Trust: The Risks of Treating AI as a Moral Partner

Sep 2026 · Philosophy & Technology · Vol 39 · 0 citations · 14 references
Ethics and Social Impacts of AI

TL;DR

The article proposes a reframing of alignment from an ethical-pedagogical project to an institutional one: instead of presuming or attempting to cultivate artificial moral agents on the basis of behavioral proxies, governance should focus on designing legal, technical, and organizational structures that constrain and render machine behavior auditable and accountable, while keeping moral responsibility firmly anchored in human agents and institutions.

Abstract

The article argues that many contemporary AI alignment practices risk a mistaken assimilation of moral agency to statistical learning. Techniques such as reinforcement learning from human feedback and constitutional AI often treat morality as a behavioral function that can be approximated from human discourse, behavior, and large-scale interaction data. Drawing on Tomasello’s account of shared intentionality, Hegel’s theory of recognition, and the second-personal tradition in moral philosophy, the article contends that moral agency is not exhausted by norm-conforming output, but presupposes participation in a space of mutual accountability, justificatory practice, and normative self-binding. On this view, and given the architecture of present generative systems, the production of normatively fluent behavior and reason-like discourse is not sufficient for moral partnership: absent interpersonal identity, recognition, and second-personal answerability, such systems can at best simulate the outer form of moral conduct. The central risk, therefore, is not that these systems behave “immorally,” but that their reliable norm-conforming behavior is misread as evidence of genuine moral commitment, encouraging misplaced trust and over-ascription of responsibility. The article proposes a reframing of alignment from an ethical-pedagogical project to an institutional one: instead of presuming or attempting to cultivate artificial moral agents on the basis of behavioral proxies, governance should focus on designing legal, technical, and organizational structures that constrain and render machine behavior auditable and accountable, while keeping moral responsibility firmly anchored in human agents and institutions.

Read PDF

Similar papers

Open access Sep 2026

A coevolutionary account of normative human–AI interaction

It is argued that the integration of aligned large language models into social practice reshapes the structure of moral agency itself, embedding derivative normative orientations that recursively feed back into human moral formation.

J. Noller · 0 citations
Open access 2026

Trusting machines with morality — Delegating moral decisions to AI

Research suggests that individuals are generally skeptical about the use of artificial intelligence (AI) in moral contexts, favoring human decision-makers over AI. Yet, in two experiments involving a total of 5639 participants, we find that individuals facing a real-life moral decision delegate significantly more often...

Nicola Hüholt, Nora Szech · 3 citations · ⚡1
Preprint Aug 2026

Incoherent by Design? On the Moral Self-Consistency of LLMs

It is argued that demonstrating internal incoherence is a necessary precursor to AI alignment as well as a broader phenomenon of epistemic instability in generative AI wherein models fail to reliably maintain coherence with respect to their own prior outputs.

Pegah Nokhiz, Aravinda Kanchana Ruwanpathirana, Helen Nissenbaum · 0 citations
Open access Aug 2026

Fear, power, and superintelligence: A realist reframing of AI catastrophic risk

Debates on catastrophic artificial intelligence (AI) risk often frame artificial superintelligence as a problem of value alignment: the central task is to ensure that advanced systems act in accordance with human intentions or moral principles. This article argues that such a framing is necessary but insufficient. The...

C. Burelli, Federico Formentini · 0 citations
Open access Sep 2026

Two Moralities of Recognition

A morality of recognition maintains that moral norms have their authority in virtue of the ways in which those norms enact an ideal moral relationship. T. M. Scanlon argues that this approach can yield an account of the distinctive force of morality and an attractive account of moral motivation. We find this approach...

Barry Maguire, Oded Na'aman · 0 citations
#artificial intelligence Preprint Oct 2026

Formalizing the Moral Evaluation of Speech Acts: Truthfulness, Lies and Ethical Dilemmas

In life-or-death situations, a benevolent lie may appear more moral than telling the truth. Yet such lies can backfire, producing unintended and sometimes fatal consequences. This tension, famously disputed by Kant and Constant in 1797, applies not only to lying but to assertive speech acts in general, raising the ques...

Benjamin Icard, Gauvain Bourgne, Jeanne Bonnaventure et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

Microsoft Research Blog Nov 3, 2026

Learn the rules: The hacker mindset

Research Manager Mike Walker reflects on a career spanning hacker culture, cybersecurity, and AI, sharing lessons about trust, teamwork, leadership, and why solving difficult problems requires curiosity, judgment, agility—and an understanding of the rules. The post Learn the rules: The hacker mindset appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.