Heng Liang

Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models

Large language models have shown strong reasoning capabilities, but their high inference costs make knowledge distillation an important approach for transferring such capabilities to compact models in resource-constrained scenarios. On-policy self-distillation further reduces the reliance on external large teacher mode...