Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer
Transformers asked to multiply multi-digit numbers in a single forward pass often fail, and interpretability studies of pretrained language models find arithmetic solved by input-range heuristics rather than by an explicit carry. We train small Llama-style transformers from scratch on 4x4 multiplication without chain o...
Sama Satariyan, Raphael Cousin, Gérard Biau
· 0 citations