Skip to content

Knowledge Editing for Masked Diffusion Language Models

Jun 2026 · 1 citation · 37 references
Computer Science

TL;DR

A simple correction is introduced that optimizes the edit including such states in which the target is partially unmasked, substantially restoring multi-token performance in masked diffusion models.

Abstract

Knowledge editing aims to update or correct factual knowledge in a language model. A widely used approach, locate-then-edit, first localizes a fact within the model and then edits the weights there. To date, such methods have been developed exclusively for autoregressive models (ARMs). Whether they work for masked diffusion models (MDMs), which model text bidirectionally and generate by iterative denoising rather than next-token prediction, remains an open question. We address it by transferring locate-then-edit to MDMs and comparing multiple MDMs with their matched ARMs. Our central finding has two parts. First, where an edit should be applied transfers between them: the same early-to-mid-layer MLP at the last subject token is most effective for both. Second, this shared location does not guarantee a shared outcome. Single-token edits succeed in both, but as targets grow longer, editing degrades far more sharply in the MDMs than in the ARMs. The failure stems from how the edited fact is generated: producing a multi-token target passes through intermediate states in which the target is partially unmasked, for which the edit was never optimized. Guided by this diagnosis, we introduce a simple correction that optimizes the edit including such states, substantially restoring multi-token performance. Our code is available at https://github.com/holi-lab/MDM-KE.

View source

Similar papers

How Should Diffusion Language Models Edit Code?

This work identifies two distinct requirements for successful editing: covering every required change and placing precise boundaries around it, and shows why strong infilling capability alone does not ensure reliable editing.

Xi-Jia Tao, Zi-Rui Liu, Shansan Gong et al. · 0 citations
#natural language process... Preprint Sep 2026

Representation-based Masked Diffusion Model

Representation-based Masked Diffusion Model (RMDM) is proposed, a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely, and significantly improves generation quality, particularly in aggressive few-step sampling regimes.

Yang Hu, Ding Huang, Xue-Yu Zhou et al. · 0 citations

On Trajectory-Aware Training for Masked Diffusion Language Models

PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.

Manuel Madeira, Amitis Shidani, Alice Bizeul et al. · 2 citations · ⚡1
#machine learning Open access Sep 2026

CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models

CoDR (Confidence Drift Remasking), a training-free and sampler-agnostic refinement pass that estimates drift for all committed positions in only k forward passes via k-partition probing, and regenerates only the tokens the model no longer endorses.

Yue Wu, Qing-He Zheng, Yu Zhang et al. · 0 citations
Review Open access Sep 2026

An overview of discrete diffusion models in natural language processing

Discrete Diffusion Models (DDMs) have recently emerged as a promising paradigm for text generation, providing an alternative to the dominant autoregressive (AR) approach. Unlike AR models that rely on sequential decoding and suffer from exposure bias, DDMs operate in discrete token space through iterative denoising, en...

Yi-Xuan Li, Kai-Dong Yu, Shuang-Yong Song et al. · 0 citations

Spectral Feedback for Test-Time Alignment of Protein Diffusion Models

Discrete diffusion models have become leading architectures for inverse protein folding. Test-time alignment guides these models during inference to generate proteins with desirable attributes. Alignment methods for discrete diffusion models often treat inference as a unidirectional process, lacking effective mechanism...

Shai Dickman, M. Cemri, Landon Butler et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.