Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models
Diffusion language models apply bidirectional attention over the whole context at every denoising step, so a longer context makes each step substantially more expensive. Sparsifying only at inference leaves the pretrained weights unchanged, so the model is never trained under the sparse pattern it runs with. We propose...
Qi Han, Jing Xiong, Zi-Yuan Zhang et al.
· 0 citations