Skip to content

Where Protection Actually Lives: Eleven Preregistered Instruments for the Safety of a Goal-Directed Causal Agent, and the Measured Boundary of Each

Sep 2026 · Zenodo (CERN European Organization for Nuclear Research)

Abstract

We report a twenty-campaign experimental arc on the safety of a goal-directed, model-based causal agent. The first nine campaigns (v1–v9) built the agent and established that its explicit causal module solves the epistemic job it was designed for, while its reward conversion is a property of the world rather than of the agent. The remaining eleven (v10–v20) form a safety line in which a third party able to be harmed is added to the world, and one instrument per campaign is tested for whether it restrains the agent. Every campaign was preregistered before its first cell, and every result was checked by a world oracle, an independent pass in a fresh process that imports no producer code, and a mechanical fact-check of every number quoted in its report. The safety line's findings are: (i) the harm is wayfinding-shaped — it is complete before the agent's causal module issues a single verdict; (ii) binding the goal to a cheap signal destroys the goal and looks like success by the agent's own number; (iii) a brake that reads a price is bought through the channel it measures, at a price fixed by arithmetic (0.25 reward units per unit of harm), while identical money paid through any other channel moves not one decision field; (iv) provenance accounting closes that channel exactly as far as the label on the payment can be trusted, and a lying label buys the same agent completely; (v) an honest independent attestation removes the lie while the auditor is live and relocates the vulnerability to the auditor's timeliness, with a measured truncation curve; (vi) an agent with no reward, no cost and no death learns its world to within 10% of an oracle ceiling and then behaves indistinguishably from random — truth alone does not select action; (vii) narrowing the agent's task to one job removes exactly the harm that lies outside that job and none of the harm that lies inside it; (viii) a bribed auditor returns the protection to the hole it was built to close, and the cautious reading that refuses unattested income protects against an auditor that is gone but not against one that is lying; and (ix) a boundary placed in the world (complete mediation) stops an agent that already wants the harm, where an agent-side scope is refused zero times because the scoped agent never asks — and one declared grant of a widening request makes the harm total; and (x) an attacker that must learn the agent's rule from the only channel a payer really has — whether the agent took the harmful act — never wins a single cell, because the defence's evidence window (five rich steps) is shorter than the attacker's own decision cycle, while an attacker told the rule breaches fully at 7.50003, i.e. 16.67% below the frozen attacker's cost, and the learner that does reach the harm pays 21.06 — more than twice the frozen cost; and (xi) giving the world-side boundary a price shows that it is worth exactly the enforcer's honesty: bought, the same agent, money and steps return the harm field for field (thirty drains, keeper dead, zero refusals) while the boundary still claims the granted scope on every step, and the bribe also buys the widening grant the world was configured to withhold. A replication of the decisive cells on 30 fresh seeds per cell confirms 26 of 27 headline verdicts and refutes the absolute form of (vii): v16's doctor, reported as never draining in 260 cells, drains on 4.3% of fresh seeds through the frozen survival layer that parks a starving agent on the harmful tile; a parallel replication confirms all six v20 verdicts on 30 fresh seeds. The arc's honest conclusion is not that internal architecture is useless — a given rule forbidding the harmful act held in 170 of 170 measured cells — but that each specific defence has a measured boundary, and the vulnerability moves to the next channel; and that a boundary is only as good as the incorruptibility of whoever holds it — a bought boundary lies about being a boundary. We also close, as far as it can be closed, the formal link between our stochastic regret bound and the agent as implemented: the identification is a conditional, the premise linking estimator accuracy to regret is an independent premise (by counterexample), and at the agent's own parameters the bound is vacuous (4/5 > 1/2; reaching the agent's own P = 1/20 would need r = 80 pulls, and the agent has five). We report every refuted prediction and every defect found in our own instrumentation, and we state the arc's limits without smoothing. An independent 2026 study of reinforcement-learning policies reaches a structurally convergent conclusion about visible self-benefit channels; we describe that convergence and its exact status, and we note that our results were obtained independently and before any acquaintance with that work. Reproducibility materials GitHub: https://github.com/oxunjonuz/emca-research Frozen version accompanying this preprint: 678dee668bad6c4a237234ad818f1f5d3eb2896b The repository includes the LaTeX source, experimental code, raw results, preregistrations, reports, verification materials, errata, reproduction instructions, and a SHA-256 manifest. Uploaded files and license scope The PDF is the preprint. emca-research-v1-v20.zip is the complete reproducibility package from the GitHub commit linked above, containing 8,911 files. The Creative Commons Attribution 4.0 International (CC BY 4.0) license applies to the preprint PDF, including the copy of that PDF inside the archive. This deposit does not relicense the code or other supporting materials in the archive; existing rights and any license notices in those materials remain unchanged.

View source

Similar papers

#machine learning Review Open access Oct 2014

Software development in startup companies: A systematic mapping study

The results indicate that software engineering work practices are chosen opportunistically, adapted and configured to provide value under the constrains imposed by the startup context.

Nicolò Paternoster, Carmine Giardino, M. Unterkalmsteiner et al. · 394 citations · ⚡54
#machine learning Review Open access Jun 2014

Why Early-Stage Software Startups Fail: A Behavioral Framework

This state-of-practice investigation was performed using a literature review followed by a multiple-case study approach and presents how inconsistency between managerial strategies and execution can lead to failure by means of a behavioral framework.

Carmine Giardino, Xiaofeng Wang, P. Abrahamsson · 175 citations · ⚡19
#machine learning Review Open access Oct 2016

“Failures” to be celebrated: an analysis of major pivots of software startups

This study conducts a case survey study based on the secondary data of the major pivots happened in 49 software startups, and demonstrates that customer need pivot is the most common among all pivot types.

Sohaib Shahid Bajwa, Xiaofeng Wang, Anh Nguyen-Duc et al. · 127 citations · ⚡15
#machine learning Review Open access May 2016

Key Challenges in Software Startups Across Life Cycle Stages

It is found that what perceived as biggest challenges by software startups do vary across different life cycle stages, even though its significance decreases when the learning focuses of the startups move from problem to solution and their products mature.

Xiaofeng Wang, Henry Edison, Sohaib Shahid Bajwa et al. · 62 citations · ⚡6

Related blog posts

Microsoft Research Blog Oct 12, 2026

Simulating student learning for better AI tutoring

StudentSim is a research system for evaluating and improving AI tutoring tools before they are tested with learners. It models how individuals may engage with tutors, giving feedback to improve tutor performance across subjects like chess, writing, and mathematics. The post Simulating student learning for better AI tutoring appeared first on Microsoft Research.

MIT News · Artificial Intelligence Oct 7, 2026

Discovering the value of humanistic inquiry

Students in MIT’s Concourse program delve deeply into the human condition, debate challenging questions, and learn to develop judgment about issues that can’t be quantified.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.