Why Knowledge Distillation Fails to Scale in Neural Retrieval
TL;DR
This work hypothesizes that the teacher model's capacity acts as an information bottleneck, limiting how much large student models can learn, and provides practical guidance for selecting appropriate teacher models when training large-scale neural retrievers with knowledge distillation.
Abstract
Knowledge distillation (KD) from cross-encoder teachers is a widely adopted technique for training effective neural retrieval models. However, recent studies have revealed a puzzling phenomenon: while retrieval models trained with contrastive loss (CL) exhibit clear scaling behavior with larger language models, KD-trained models show minimal performance gains as model size increases from 1B to 8B parameters. The underlying cause of this scaling failure remains unexplored. In this work, we hypothesize that the teacher model's capacity acts as an information bottleneck, limiting how much large student models can learn. To test this hypothesis, we conduct systematic experiments using decoder-only LLMs (Llama-3: 1B, 3B, 8B) as student retrievers and cross-encoder teachers ranging from 66M to 3B parameters. Our experiments on MSMARCO and BEIR benchmarks reveal that: (1) Small teachers severely constrain student scaling, with 1B, 3B, and 8B students performing nearly identically. (2) Larger teachers progressively restore scaling behavior, enabling significant performance gains at the 8B scale. (3) A teacher-to-student parameter ratio above a critical threshold appears necessary for effective knowledge transfer. Our findings provide practical guidance for selecting appropriate teacher models when training large-scale neural retrievers with knowledge distillation.