Jul 2026
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
The results show that standard transformers are rarely a local optimum in the space of architectures, and suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.
Damien Teney, Liangze Jiang, Hemanth Saratchandran et al.
· arXiv.org · 2 citations