Preprint

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games

Ilia Karpov

Jul 2026 · 1 citation · 57 references

Computer Science

TL;DR

MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind, is presented, finding that stated confidence is poorly calibrated, agents overestimate how often they are suspected by a factor of 1.5, and single-vote counterfactual replays rarely change game outcomes.

Abstract

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. It distinguishes whether an agent lost because it misread the game or because it failed to act on a correct assessment, a distinction that is invisible from outcomes and dialogue transcripts alone. After every public utterance, each agent privately answers structured probe questions whose responses never re-enter the game and are scored against the ground truth known to the engine. An interactive visualizer replays games from the perspective of an individual agent's beliefs, displays timeline-aligned accuracy and calibration, and supports counterfactual replay from any recorded step. In a case study across two model families comprising tens of thousands of parsed probe responses, we find that stated confidence is poorly calibrated, agents overestimate how often they are suspected by a factor of 1.5, and single-vote counterfactual replays rarely change game outcomes: outcome flips occur primarily when the agent had already formed a correct belief state, whereas decisions made under an incorrect model of the world remain largely unchanged under resampling. The engine, visualizer, recorded games, and counterfactual replay corpus are released under an open-source licence. Code: https://github.com/karpovilia/mafiascope. Live demo: https://karpovilia.github.io/mafiascope/. Screencast: https://vimeo.com/1208920221.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives

A knowledge-verified benchmark that first confirms through a neutral probe that an agent knows a user's entitlement, and then evaluates whether it makes false claims once an incentive to deny that entitlement is introduced, which reduces the confound between lying and not knowing and enables more rigorous auditing and steering of agent honesty.

Zheyuan Liu, Weiliang Zhao, Xiangchi Yuan et al. · 0 citations

Review Jul 2026

Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games

An auditable framework is built that maintains an external belief state over hidden roles, logs belief updates and belief-action deviations as structured evidence, and supports a defensive offline improvement loop that reviews bad cases before any strategy change.

Yuanpeng Gao, Jiangyi Yang, Yao Zhao et al. · arXiv.org · 0 citations

Preprint Aug 2026

Confident at the moment of action: belief miscalibration in LLM play under hidden information

This work tests a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces, and where an agent's stated probability distribution over the opponent's hidden royal piece -- elicited every turn, separately from the move it chooses -- is scored against ground truth recoverable after the game.

Bhushan Kashinath Joshi · 0 citations

Jul 2026

Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game

As large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems, understanding their deceptive capabilities is fundamental to safety. Controlled social deduction games provide a reproducible proxy for isolating and evaluating these complex adversarial behaviors. We present the open-source benchmark framework ParliamentBench based on the game Secret Hitler to evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry. We evaluate 16 LLMs across 1,600 simulated matches playing each other, playing against humans, and compare them against a large set of online games. We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency. Our experiments reveal that frontier models achieve strong performance across cooperative and deceptive roles, with a strong top-four cluster (GPT-5.4, Kimi K2.5, Grok 4.1 Fast, and DeepSeek 3.1 Terminus), whereas the weakest models fall short of random (33%) and simple algorithmic (45%) baselines. Most LLMs struggle to maintain a consistent deceptive persona throughout an entire game, with deception retention dropping below 50%.

Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa et al. · arXiv.org · 0 citations

Preprint Jul 2026

When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games

Evaluating three frontier models across six games in homogeneous and heterogeneous groups over 10 rounds, it is found that different models interpret announcements incompatibly, some as binding commitments and others as cheap talk, producing payoff gaps that emerge in Round~0 and persist across all 10 rounds.

Jerick Shi, Terry Jingchen Zhang, Bernhard Scholkopf et al. · 0 citations

Preprint Aug 2026

Level-k Distinguishable Mechanisms for Evaluating Bounded Rationality in LLMs

This work formalises a necessary level-K distinguishability condition for strategic depth inference and builds a suite of novel game structures that meet this standard, and finds that model models maintain accurate strategic depth under recursive reasoning, with strong internal consistency between stated reasoning and actions at every level.

Binchi Zhang, Atrisha Sarkar · 0 citations