What Is Superintelligence?

Abstract

A machine may reason far beyond human ability, recognize that its own objective should be revised, and still continue pursuing it because that judgment has no governing authority. What does “superintelligence” tell us about such a machine? This paper distinguishes downstream capability—how effectively a system pursues an objective—from governing intelligence: reasoning whose relevant judgments can become authoritative over the commitments directing the system’s activity. The analysis identifies four routes by which a governing commitment can remain unchanged indefinitely: non-reflection at the governing layer (the Artificial Calculator), reflection without governing authority (governance lock), permanent closure of a previously available reflective path (recursive reasoning halt), and repeated retention through live judgment (judgment-guided retention). The same stable goal can therefore persist through fundamentally different governing architectures. In the first three routes, the commitment is architecturally review-immune: no future relevant judgment can govern its continued role. In the fourth, it remains stable because continuing judgment supports it. This distinction clarifies what goal stability leaves unresolved in the orthogonality debate. A reflective self-governing agent may retain its goals throughout its lifetime. What matters is whether relevant judgment can govern their retention or revision. Applied to a review-immune paperclip maximizer, the framework yields a structural reclassification: extreme downstream capability combined with a governing-intelligence downgrade relative to an equally capable self-governing architecture. This identifies an architectural problem for AI safety: increasing capability can coexist with restrictions that prevent reasoning from correcting errors in the commitments directing that capability. The same framework explains how deference to human operators or institutions can form part of reflective self-governance. Under reviewable deference, authorized human intervention can redirect, suspend, or override the system within the delegated domain, while the governing arrangement remains open to relevant evaluation. Corrigibility and reflective agency can thus operate within the same effective governing loop. The argument then extends from goals to control and final authority. Control changes what other agents can do, which evidence becomes available, and which future options can emerge. Humans, other AI systems, and scientific communities may generate contributions for which the controller lacks a complete substitute. Irreversibly removing such a source can eliminate a path through which the controller’s own error would become visible. Exhaustive calculation within its current model does not, by itself, establish that the model covers every relevant possibility or supply the evidence needed to determine which possibility is actual. These effects make the continued form of authority a potential object of governing judgment. Under the paper’s open-world conditions, architecturally review-immune final authority cannot persist as the continuing output of reflective self-governance at that same layer. Authority may nevertheless remain unchanged indefinitely through judgment-guided retention. Where other agents’ options depend on review-immune authority that is also arbitrary or uncontrolled in the republican sense, the resulting relation can be understood as domination. Six explicit failure conditions identify how the functional distinctions and classification could be challenged, and when the causal or practical rationale for authority review would require revision. Across goals, deference, and final authority, the paper develops one diagnostic question: is the intelligence doing the thinking also the intelligence governing the machine? Keywords: superintelligence; instrumental reasoning; governing intelligence; reflective intelligence; reflective self-governance; fixed goals; orthogonality thesis; AI safety; Artificial Calculator; governance lock; recursive reasoning halt; final authority; review-immunity; corrigibility; domination