Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
Though the construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments it is demonstrated that the transformer variant can learn and generalize perfectly on other common length generalization benchmarks, including modular arithmetic and ListOps.
Takuya Ito, Ruchir Puri, Murray Campbell et al.
· 0 citations