Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
Though the construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments it is demonstrated that the transformer variant can learn and generalize perfectly on other common length generalization benchmarks, including modular arithmetic and ListOps.