Exploring the Potential of Diffusion Large Language Models in Code Generation

Abstract

Large Language Models (LLMs) have revolutionized the landscape of code generation. Existing LLMs mainly employ autoregressive generation, i.e., generating code token-by-token from left to right. However, the underlying autoregressive generation has two limitations in code generation. First, autoregressive LLMs typically generate only one token per step, showing low efficiency in practice. Second, programming is a non-sequential process involving back-and-forth editing, while autoregressive LLMs only employ the left-to-right generation order. These two intrinsic limitations hinder the further development of LLMs in code generation. Recently, diffusion LLMs have emerged as a promising alternative. They provide two characteristics that are particularly relevant to code generation: multi-token prediction (i.e., generating multiple tokens at each step) and flexible generation order (i.e., flexibly determining which positions to generate tokens). However, there is no systematic study exploring diffusion LLMs in code generation. To bridge the knowledge gap, we present the first empirical study of diffusion LLMs for code generation. Our study involves 7 representative diffusion LLMs and conducts experiments on a wide range of benchmarks. Based on the results, we summarize the following findings. ❶ Current diffusion LLMs show promising but uneven code generation ability, with open-source diffusion LLMs becoming competitive in several settings and closed-source diffusion LLMs showing stronger results. For example, on MBPP+, the best-performing diffusion LLM achieves a pass@1 score of 79.1%, compared with 73.3% for the best evaluated autoregressive baseline. ❷ Diffusion LLMs exhibit stronger length extrapolation behavior in our long-code-understanding experiments. ❸ We explore factors impacting the effectiveness and efficiency of diffusion LLMs, and provide practical guidance. ❹ We discuss several promising future directions to improve diffusion LLMs on code generation. We open-source all source code, data, and results to facilitate future research.