This work proposes a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation and derives a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs.
Abstract
Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.
Representation-based Masked Diffusion Model (RMDM) is proposed, a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely, and significantly improves generation quality, particularly in aggressive few-step sampling regimes.
Yang Hu, Ding Huang, Xue-Yu Zhou et al.· 0 citations
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
Enhanced Mixture-of-Experts (E-MoE) is proposed, which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline.
Arseny Ivanov, A. Kolesov, Alexander Korotin et al.· arXiv.org· 1 citation
We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature space of a frozen pretrained DLM. To estimate MMD, we retain contextual features at individual token positions, obtaining multiple observati...
Ilya Drobyshevskiy, I. Sudakov, Maksim Semenov et al.· 0 citations
Prefix-Conditioned Diffusion (PCD), which samples a boundary, preserves the prefix, preserves the prefix, and denoises the suffix, is proposed, and improvements in reasoning and coding over native diffusion training are shown.
Xiao-Cheng Lu, Hua-Bin Liu, Song Guo et al.· 0 citations
Less Uniform Diffusion (LUDI) is proposed, a novel UDLM framework that introduces a less uniform loss that directs each reverse transition toward the clean token, and equip the model with per-token time embeddings that supply token-level corruption hints, enabling confidence-based few-step sampling.
Kai-Bo Wang, Ding Ding, Fang-Yuan Ding et al.· Proceedings of the 2026 Conf...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.