RAPID: Reliability-Aware Pair Importance Distillation
Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing all pairs has quadratic complexity in the batch size, whereas uniform subsampling may use a limited relation budget inefficiently. We introduce Reliability Aware Pair I...
Ali Mahdavi, A. Zamanifar, Amirfarhad Farhadi et al.
· 0 citations