Somtochukwu Anunobi

Architectural Inductive Bias in Knowledge Distillation: Disentangling Representational Similarity from Performance Transfer

Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s saf...