Superintelligent Zombies

Abstract

We explore and argue for the view that intelligence beyond a certain threshold is negatively correlated with consciousness, so that the more intelligent a system is, the less likely it is to be conscious. ‘Intelligence is the enemy of consciousness’, as we put it. This view entails that superintelligent AI models are likely to be zombies. After presenting an initial defence of the view that draws on the research programme of resource-rational analysis, we argue that recent developments in AI research provide additional evidence supporting it. We conclude by exploring its moral implications. First, we suggest that if our view is right, a certain type of moral risk can be averted since researchers are unlikely to stumble into accidentally creating AI models that are conscious. Second, we discuss a new form of AI risk that arises from our view.