An empirical study of teacher supervision failure in adversarial distillation for lightweight visual recognition

Abstract

Adversarial distillation transfers the robustness of a strong teacher to a lightweight student, and most methods implicitly assume the teacher is reliable at every adversarial point. We test this assumption empirically. We define teacher supervision failure as the event that the teacher’s top-1 prediction on an adversarial example disagrees with the ground-truth label, and study it at the sample level on 200 PGD-10 examples on CIFAR-10 with a WideResNet-34-10 teacher and a ResNet-18 student. We quantify how often the teacher fails, characterize the failure regime through confidence and decision margin, and analyze when a reliability-gated correction still yields a correct student prediction. The teacher fails on 44% of examples; failures occupy a distinct low-confidence, low-margin regime; and on failure cases the corrected student is correct 23.9% of the time, concentrated where the teacher is only weakly wrong. These results motivate reliability-aware supervision in adversarial distillation.