Skip to content

Looped Transformers as Optimizers

Sep 2026 · arXiv.org · Vol abs/2609.37379 · 0 citations · 57 references
Computer Science

TL;DR

This framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule, which combines explicit weight decay, adaptive step size and a delta objective and improves average generative performance over the compared looped and non-looped baselines under matched training FLOPs.

Abstract

Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models have likewise highlighted the value of scaling test-time computation through longer computation trajectories. However, the principles for designing effective loop transitions remain poorly understood. We view the looped hidden state as a fast weight that is updated throughout the depth. We formulate loop transitions as local gradient-based updates, with recurrent blocks predicting implicit targets at each depth. Our framework derives loop transitions in closed form from a projection, a local objective and an optimizer update rule. Mapping representative loop transitions into this framework reveals mismatches between their transitions and projections. We first align the input maps of existing transitions. We then derive OperLoop, which combines explicit weight decay, adaptive step size and a delta objective. The aligned variants reduce training loss and improve average commonsense accuracy. OperLoop improves average generative performance over the compared looped and non-looped baselines under matched training FLOPs. These results support the framework's usefulness for loop design. We extend the analysis to additional loop models and outline a roadmap for future loop transition design.

View source

Similar papers

Improving Test-Time Scaling with Adaptive Looped Transformers

Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains und...

Yi-Chen You, Tianyu Fu, Ao-Song Feng et al. · 4 citations

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

This work proposes LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers that introduces a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes.

SangLyul Cho, Lang-Qing Cui, Sehoon Kim et al. · 7 citations · ⚡1
#machine learning Preprint Oct 2026

Decoding Looped Transformers Better for (Almost) Free

Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies...

Weihao Liu, Huangjie Zheng, Tian-Rong Chen et al. · 2 citations
#artificial intelligence Preprint Oct 2026

LiFT: Loop Flow Transformers

We introduce Loop Flow Transformers (LiFT), a family of looped generative models that scales computation by repeatedly applying a shared Diffusion Transformer (DiT) core, with only light changes to the standard architecture. Rather than asking every recurrent step for the final prediction, LiFT trains each step with a...

M. Derakhshani, Pedro M. P. Curvo, G. Burghouts et al. · 1 citation

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.