Jul 2026· IEEE International Conference on Circuits and Systems for Communications· pp. 1-7· 0 citations· 18 references
Abstract
Deep neural networks often achieve high accuracy, but their compute, memory, and power demands can hinder deployment on resource-constrained edge devices. This paper proposes an iterative, constraint-aware knowledge distillation framework that compresses a large model into a small one under an explicit model-size budget and a maximum allowable accuracy drop. Starting from user-specified constraints, the method trains candidate students, checks feasibility, and adjusts capacity across generations; oversized students may be promoted to intermediate teachers for subsequent distillation stages. Experiments on MNIST and Speech Commands validate the approach: on MNIST, a 100 KB target with a 2-point accuracy-drop budget yields a 55 KB student achieving 97.28% accuracy; on Speech Commands, a 4-point drop yields a 197.5 KB student reaching 94.08% accuracy. These results demonstrate predictable tradeoffs between compactness and accuracy, and show that multistage teacher promotion can bridge large capacity gaps while preserving performance within user-defined tolerances.
This article introduces variance-reduced consistency learning (vrCL), a novel distillation technique that enables stable and efficient training of consistency models without relying on teacher model evaluations, resulting in high computational efficiency and significantly reduced training time.
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s saf...
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations
Shift-Augmented Knowledge Distillation (SAKD) is proposed, a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation, enabling single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift.
Xuyi Yu, Yaohua Liu, Chengjun Li et al.· 0 citations
This paper presents Poly-OPD, a framework that can consolidate complementary strengths of heterogeneous teachers into a single compact flow-matching student, and uses a gradient compatibility diagnostic to organize its adapters.
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner et al.· 0 citations
A practitioner's study of how to make distillation training efficient is presented, organised around two systems contributions, and a fused, chunked KL loss is introduced, making peak memory linear in the sequence length.
Bakbergen Ryskulov, Iker García-Ferrero, David Montero et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
