Dual-Vocabulary Language Model for Cross-Tokenizer Distillation
On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grain...
Ke-Di Chen, Chen Lin, Yutao Sun et al.
· 1 citation