Skip to content

UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute

Oct 2026 · 0 citations · 41 references
Computer Science

TL;DR

UpLoop is a pretraining schedule that uses upcycling to train looped Transformers while exactly matching a standard dense baseline in final parameter count, total training tokens, and training FLOPs, showing improvements in language modeling and downstream performance.

Abstract

Looped language models can strengthen reasoning through repeated computation with shared weights, increasing execution depth without adding independent parameters. However, each additional pass increases training floating-point operations (FLOPs), making these gains more expensive to obtain. To address this cost, we propose UpLoop, a pretraining schedule that uses upcycling to train looped Transformers while exactly matching a standard dense baseline in final parameter count, total training tokens, and training FLOPs. Specifically, UpLoop begins with a shallow Transformer, progressively grows its depth by copying learned layers, and uses the computation saved early to support repeated execution of a shared middle block later in training. Experiments across model scales show improvements in language modeling and downstream performance, with the average score across 22 downstream tasks increasing from 22.99 to 27.52 at 1.3B parameters. These gains also extend to mathematical reasoning after supervised fine-tuning.

View source

Similar papers

Preprint Oct 2026

UpLoop: Upcycling Large Language Models with Gated Loop

Looped large language models trade parameter storage for recurrent computation, but most are trained from scratch. In practice, pretraining with looped depth is expensive, making post-training upcycling of a pretrained non-looped LLM an attractive alternative. We introduce \textbf{UpLoop}, a post-training recipe based...

Tie-Yuan Chen, Hua-Bin Liu, Ying-Tao Zhang et al. · 0 citations
#natural language process... Preprint Oct 2026

LoopFold: Parallelizing Recurrent Depth in Looped Transformers

Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. E...

Fei Song, Bo-Yi Zeng, Yi-Qing Hao et al. · 0 citations
#artificial intelligence Preprint Oct 2026

LiFT: Loop Flow Transformers

We introduce Loop Flow Transformers (LiFT), a family of looped generative models that scales computation by repeatedly applying a shared Diffusion Transformer (DiT) core, with only light changes to the standard architecture. Rather than asking every recurrent step for the final prediction, LiFT trains each step with a...

M. Derakhshani, Pedro M. P. Curvo, G. Burghouts et al. · 1 citation
#artificial intelligence Preprint Sep 2026

Closing the Loop: Practical Training Recipes for Looped Language Models

Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practic...

Andrei Marchenko, В. Е. Безруков, Oleg Kashurin et al. · 3 citations · ⚡1

Improving Test-Time Scaling with Adaptive Looped Transformers

Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains und...

Yi-Chen You, Tianyu Fu, Ao-Song Feng et al. · 4 citations
#machine learning Preprint Nov 2026

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later...

Xuan Ouyang, Ze-Fan Cai, Chen-Long Zhang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.