This work investigates whether hybrid backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, and finds that they can be efficient starting points for adaptation.
Abstract
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.
ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.
Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al.· 1 citation
Zarya is introduced, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.
Leonid S. Sinev, Ilya Koziev, Vladislav Leshchuk· arXiv.org· 0 citations
Discrete Diffusion Models (DDMs) have recently emerged as a promising paradigm for text generation, providing an alternative to the dominant autoregressive (AR) approach. Unlike AR models that rely on sequential decoding and suffer from exposure bias, DDMs operate in discrete token space through iterative denoising, en...
Yi-Xuan Li, Kai-Dong Yu, Shuang-Yong Song et al.· Vicinagearth· 0 citations
Temporal Self-Distillation (TSD), a simple on-policy method that trains dLLMs for fast inference by distilling predictions across time, substantially shifts the speed--quality frontier toward the low-compute regime.
Shi-Jia Xu, Andrea Miele, Metod Jazbec et al.· 0 citations
A2D, a simple training-free framework for enhancing diffusion models with existing AR post-training resources, is proposed and can transfer capabilities from AR post-trained models to diffusion base models, and further improve already post-trained diffusion models by composing AR and diffusion post-training updates.
Yi-Ming Qin, Ke Wang, Amel Abdelraheem et al.· 0 citations
Pretrained autoregressive Transformers represent a large sunk investment in compute. Existing conversion methods reuse that investment by changing either the architecture (attention to recurrence) or the objective (next-token prediction to denoising), never both. We convert Qwen3 teachers at 1.7B and 8B into attention-...
Julian Boesch, Andrew Wee, Alexander Stranzl· 1 citation· ⚡1
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.