A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions
This review presents a unified Bayesian formulation of KD that formulates teacher predictions as prior information and establishes a rigorous connection to uncertainty quantification, providing a principled interpretation of how teacher information is incorporated into student learning.
Lu-Yang Fang, Hao-Ran Lu, Jia-Zhang Cai et al.
· 0 citations