Skip to content

Recursive Scaling in Masked Diffusion Models

Jun 2026 · arXiv.org · Vol abs/2606.18022 · 2 citations · 48 references
Computer Science

TL;DR

Recursion in MDMs is introduced and new Recursive Masked Diffusion Models (R-MDMs) are proposed, which apply a shared denoising transformer several times within each denoising step, adding recursive depth as an additional compute axis without increasing parameter count.

Abstract

Masked diffusion models (MDMs) generate sequences by iteratively refining a partially masked state and committing tokens in parallel. We introduce recursion in MDMs and propose new Recursive Masked Diffusion Models (R-MDMs), which apply a shared denoising transformer $L$ times within each denoising step, adding recursive depth as an additional compute axis without increasing parameter count. Across structured generation tasks, recursive depth improves quality at fixed parameter budget, matches substantially larger non-recursive models at matched FLOPs, and can reduce the number of denoising steps needed to reach a target quality. We interpret these gains with a dependence--fidelity decomposition of parallel decoding error: recursion refines model marginals at a fixed masked state, whereas denoising steps change that state by committing tokens. Building on this analysis, we propose to treat decoding as a two-axis decision (how many loops to run and which tokens to commit) and show that entropy-guided adaptive rules improve the quality--compute frontier over fixed schedules, transferring across various tasks on Sudoku, Countdown, RNA, and executable math generation. Together, these results establish recursive depth as a practical, complementary test-time scaling mechanism for MDMs.

View source

Similar papers

#machine learning Preprint Oct 2026

Random Remasking Scales Masked Diffusion Inference

Masked diffusion models (MDMs) generate sequences by progressively unmasking tokens, but errors introduced during decoding are typically irreversible. Recent work has shown that iterative correction can improve sample quality, yet existing theory fails to explain this given a learned, imperfect denoiser model. We re-fo...

Zhao-Yi Zhou, F. Wu, Andrea Zanette · 0 citations
#machine learning Preprint Oct 2026

LED: Improving Masked Diffusion Language Models by Lookahead Editing

This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs, and fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-\texttt{7B} and LED-\texttt{8B}, respectively.

Yu-Yan Zhou, Ze-Yuan Yang, Yuxuan Song et al. · 0 citations
#machine learning Preprint Oct 2026

Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

This work establishes end-to-end statistical guarantees showing that transformer-based MDMs learn structured distributions with a sample complexity governed by their latent structure through mixtures of hidden Markov trees, which capture latent dependencies within components and heterogeneity across components.

Jing-Da Wu, Chang-Xiao Cai · 1 citation

Zarya: A Hybrid Autoregressive-Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference

Zarya is introduced, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.

Leonid S. Sinev, Ilya Koziev, Vladislav Leshchuk · 0 citations
#machine learning Preprint Oct 2026

Multi-Stream Residual Computation in Diffusion Language Models

Diffusion language models reconstruct masked tokens in parallel, offering an alternative to autoregressive generation. Yet their Transformer backbones typically carry visible evidence and evolving masked predictions through a single residual stream. We equip masked diffusion models with multiple residual streams that p...

Ekaterina Alimaskina, S. Kazarian, G. Molodtsov et al. · 0 citations
Preprint Oct 2026

TDLM: Adaptive Temporal Integration for Diffusion Language Models

Temporal Self-Conditioning Diffusion Language Model (TDLM) is introduced, which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser, which gives later predictions access to computation already performed.

Bo-Han Sun, Max Zhu, Francesco Caso et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.