Preprint

Allocating Recurrent Compute in Looped Language Models

Ruhai Lin Yi-Yang Guo Ruiming Zhu H. Ye Jason K. Eshraghian

Aug 2026 · 4 citations · 25 references

Computer Science

TL;DR

Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.

Abstract

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout. Iterative Transport Rank (ITR) describes the cumulative influence trajectory; marginal ITR describes the nonredundant influence contributed by successive applications. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. We compare MixerLoop with no recurrence and full-block recurrence at 15M and 110M parameters under the same data, initialization, and architecture. A finite context-off intervention tests whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language-model readout. MixerLoop surpasses FullLoop on aggregate CORE at 15M and retains 41.5% of its CORE improvement at 110M while reducing recurrent-backbone projection FLOPs by 45.9%. These results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN.

arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

Looped State-Space Language Models with Adaptive Exit-State Selection

Looped Mamba and Looped Hybrid Mamba-Transformer architectures, which repeatedly apply a shared Mamba block to introduce explicit finite-depth recurrent computation, are investigated and adaptive exit-state selection improves downstream performance at intermediate depths, while actual inference-time savings require add...

Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo et al. · 0 citations

#artificial intelligence Preprint Aug 2026

Looped Language Models Improve Compositional Tool Calling

This work evaluates native and retrofitted looped language models on API-Bank, BFCL, and NESTful, comparing looped and non-looped models trained under matched supervised fine-tuning recipes and varying recurrent depth at inference time to suggest that looped language models are a promising architecture for agentic syst...

A. C. Popescu, Haitz Sáez de Ocáriz Borde, Pietro Liò · 2 citations

#machine learning Preprint Aug 2026

Dynamic Compression in Recurrent Networks

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise acr...

Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal · 0 citations

Preprint Aug 2026

Chained Recursive Language Models for Multi-Iteration Reasoning

This work proposes Chained Recursive Language Models (Chained RLM), an inference-time architecture, in which the same underlying model is called repeatedly as a sequence of fresh reasoning roots, and studies when fresh-context artifact continuation gives a measurable gain in accuracy over direct LLM answering even with...

Purbesh Mitra, S. Ulukus · 0 citations

Preprint Jul 2026

SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode, and is orthogonal to token-level speculative decoding.

Huajun Bai, Weiwei Lv, Huichuan Zheng et al. · 1 citation

Jul 2026

TransMem: Transforming Hidden States into Memory for Large Language Models

TransMem is proposed, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations and introduces evidence-conditioned self-distillation to learn transferable memory utilization rather than task-specific knowledge.

Haodong Lei, Junming Liu, Yirong Chen et al. · arXiv.org · 0 citations