Chang Liu

How Should Reasoning Be Organized in a Transformer's Latent Space?

Experiments with two-layer and GPT-2 Transformers reproduce the predicted width advantage and show that unequal weights fail first on the states that receive the least weight, suggesting a important principle: keep reached states equally weighted, and restore equal weights as computation proceeds.

How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superposition

Large language models solve hard problems through intermediate computations across multi-step reasoning. Traditional chain-of-thought encodes these computations as tokens. Recent continuous and recurrent methods instead move partial computations into fixed-dimensional latent states, where a single thought can superpose...

Deep disagreement, fact, and moral normativity: a Wittgensteinian therapy on metaethics