A simple correction is introduced that optimizes the edit including such states in which the target is partially unmasked, substantially restoring multi-token performance in masked diffusion models.
Abstract
Knowledge editing aims to update or correct factual knowledge in a language model. A widely used approach, locate-then-edit, first localizes a fact within the model and then edits the weights there. To date, such methods have been developed exclusively for autoregressive models (ARMs). Whether they work for masked diffusion models (MDMs), which model text bidirectionally and generate by iterative denoising rather than next-token prediction, remains an open question. We address it by transferring locate-then-edit to MDMs and comparing multiple MDMs with their matched ARMs. Our central finding has two parts. First, where an edit should be applied transfers between them: the same early-to-mid-layer MLP at the last subject token is most effective for both. Second, this shared location does not guarantee a shared outcome. Single-token edits succeed in both, but as targets grow longer, editing degrades far more sharply in the MDMs than in the ARMs. The failure stems from how the edited fact is generated: producing a multi-token target passes through intermediate states in which the target is partially unmasked, for which the edit was never optimized. Guided by this diagnosis, we introduce a simple correction that optimizes the edit including such states, substantially restoring multi-token performance. Our code is available at https://github.com/holi-lab/MDM-KE.
This work identifies two distinct requirements for successful editing: covering every required change and placing precise boundaries around it, and shows why strong infilling capability alone does not ensure reliable editing.
Xi-Jia Tao, Zi-Rui Liu, Shansan Gong et al.· arXiv.org· 0 citations
Representation-based Masked Diffusion Model (RMDM) is proposed, a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely, and significantly improves generation quality, particularly in aggressive few-step sampling regimes.
Yang Hu, Ding Huang, Xue-Yu Zhou et al.· 0 citations
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
CoDR (Confidence Drift Remasking), a training-free and sampler-agnostic refinement pass that estimates drift for all committed positions in only k forward passes via k-partition probing, and regenerates only the tokens the model no longer endorses.
Yue Wu, Qing-He Zheng, Yu Zhang et al.· Conference on Empirical Meth...· 0 citations
Discrete Diffusion Models (DDMs) have recently emerged as a promising paradigm for text generation, providing an alternative to the dominant autoregressive (AR) approach. Unlike AR models that rely on sequential decoding and suffer from exposure bias, DDMs operate in discrete token space through iterative denoising, en...
Yi-Xuan Li, Kai-Dong Yu, Shuang-Yong Song et al.· Vicinagearth· 0 citations
Discrete diffusion models have become leading architectures for inverse protein folding. Test-time alignment guides these models during inference to generate proteins with desirable attributes. Alignment methods for discrete diffusion models often treat inference as a unidirectional process, lacking effective mechanism...
Shai Dickman, M. Cemri, Landon Butler et al.· arXiv.org· 0 citations