Generative modeling of discrete data has two popular paradigms: autoregressive models such as GPT, which factorize a sequence in a left-to-right (L2R) order, and masked diffusion models, which are essentially any-order autoregressive modeling. Rather than competing methods, these are the two extreme points of a single...
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In...
Wei-Jian Luo, Yu-Jian Chen, Hao-Chen Jiang et al.· 0 citations
Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query-key d...
Wei-Qi Sun, Qian-Yue Wu, Wen-Jia Wang et al.· 0 citations
Conditional Residual Prediction (CRP), a simple recipe for reducing a model's reliance on a condition: the model first predicts the target without the condition, and the condition may only add a residual on top of this prediction.
Bo-Wen Zheng, Zhi-Guang Liu, Jia-Rong Ou et al.· 0 citations
It is shown decoder-only MDMs, despite a larger modeling space, can achieve significant inference speedups and comparable perplexity with techniques like temperature annealing, offering a path to reduced inference compute.
Shuchen Xue, Tianyu Xie, Tianyang Hu et al.· arXiv.org· 17 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
