Distilling large language models for code generation via ranking supervision.

Abstract

In recent years, large language models (LLMs) have achieved remarkable advances in code generation. However, their massive parameter scales hinder deployment in resource-constrained environments. Knowledge distillation has emerged as an effective compression technique that transfers knowledge from a large teacher model to a smaller student model, thereby reducing computational cost while retaining strong generative capability. However, traditional distillation methods usually depend on forward and reverse Kullback-Leibler (KL) divergence, aligning the probability distribution over the entire vocabulary. This process makes them susceptible to long-tail noise and often leads to weaker performance than supervised fine-tuning with labeled data. To address this issue, we propose a distillation approach based on ranking supervision. At each step, the method selects the candidate tokens with the highest probabilities from the teacher's output and applies a ListNet-based loss. This loss encourages the student to learn the teacher's ranking preferences. Unlike conventional KL distillation, ranking distillation avoids exhaustive alignment of low-confidence tokens, achieving comparable training time while significantly reducing GPU memory consumption. We conduct systematic evaluations on four public benchmarks (HumanEval, MBPP, DS-1000, and MultiPL-E). Experimental results demonstrate that the proposed method consistently outperforms supervised fine-tuning as well as FKL and RKL baselines in Python code generation, multilingual generation, and data-science scenarios. Moreover, it maintains stable performance gains across different model scales, including both the Qwen2.5-Coder and DeepSeek-Coder families. Our method provides an effective solution for distilling large language models in code generation and offers guidance for future research in model compression.