Jul 2026
PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target, position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.
Zhengtao Yao, Runhao Li, Xupeng Chen et al.
· arXiv.org · 0 citations