Skip to content

Multi-Mask Diffusion Language Models for Few-Step Generation

Jul 2026 · arXiv.org · Vol abs/2607.19686 · 0 citations · 68 references
Computer Science

TL;DR

This work proposes a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation and derives a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs.

Abstract

Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.

View source

Similar papers

#natural language process... Preprint Sep 2026

Representation-based Masked Diffusion Model

Representation-based Masked Diffusion Model (RMDM) is proposed, a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely, and significantly improves generation quality, particularly in aggressive few-step sampling regimes.

Yang Hu, Ding Huang, Xue-Yu Zhou et al. · 0 citations

On Trajectory-Aware Training for Masked Diffusion Language Models

PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.

Manuel Madeira, Amitis Shidani, Alice Bizeul et al. · 2 citations · ⚡1

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

Enhanced Mixture-of-Experts (E-MoE) is proposed, which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline.

Arseny Ivanov, A. Kolesov, Alexander Korotin et al. · 1 citation
#machine learning Preprint Oct 2026

Representation-Space MMD for Diffusion Language Models

We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature space of a frozen pretrained DLM. To estimate MMD, we retain contextual features at individual token positions, obtaining multiple observati...

Ilya Drobyshevskiy, I. Sudakov, Maksim Semenov et al. · 0 citations
#artificial intelligence Conference Open access Sep 2026

Less Uniform Discrete Diffusion is More Powerful and Scalable

Less Uniform Diffusion (LUDI) is proposed, a novel UDLM framework that introduces a less uniform loss that directs each reverse transition toward the clean token, and equip the model with per-token time embeddings that supply token-level corruption hints, enabling confidence-based few-step sampling.

Kai-Bo Wang, Ding Ding, Fang-Yuan Ding et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.