Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
In in-context Mealy-machine execution, where a simulator supplies intermediate targets and exact counterfactuals for hidden-state interventions,Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with.
Pei-Long Zhou, Hong-Tao Zhang, Yu-Long Huang et al.
· 0 citations