It is shown that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation, and a method is proposed that generates teacher-specific, improved samples optimized for data-free distillation.
Abstract
Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.
It is proved that the learning dynamics and the distillation error $\Ets$ are exactly invariant to $\dmiss$, whereas the true error $\Etzs$ and the gap $\Delta=\Etzs-\Ets$ are strictly increasing in $\dmiss$, with a rate that is amplified linearly by the complexity $M_0$ of the true teacher.
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner et al.· 0 citations
Knowledge distillation (KD) has become a pivotal technique for transferring knowledge from large-scale teacher models to lightweight student models. However, traditional feature-based distillation methods necessitate the direct exposure of the teacher’s intermediate representations, raising concerns regarding data privacy and the leakage of proprietary model details. These concerns often hinder the deployment of distillation in collaborative or cloud-based scenarios. To address these challenges, we propose a privacy-aware teacher-oriented projected feature distillation framework that reduces direct feature exposure while preserving distillation effectiveness. Specifically, our method employs a low-rank projection strategy to obfuscate the teacher’s features into a compact subspace. Authorized student models leverage this projection matrix to align their own features, enabling effective knowledge transfer while reducing direct exposure of the teacher’s original spatial patterns. Extensive experiments on benchmark datasets demonstrate empirical feature obfuscation and reconstruction resistance while maintaining competitive performance in object detection and semantic segmentation, with gains in several settings even when learning from projected representations.
Jun-Fei Yi, Sihao Lin, Hui Zhang et al.· IEEE Transactions on Image P...· 0 citations
Scaling model-generated data is usually viewed as improving distillation: more examples should increase coverage, reduce noise, and produce stronger students. We show a second effect: larger datasets can make subtle teacher-specific signals easier to detect in the trained student, even when examples are off-task and never mention the trait. In a controlled setup inspired by subliminal learning, a teacher induced to express a target trait generates restricted off-task data, such as number-only completions. Students trained on different amounts of independent off-task data are evaluated in a separate domain, with matched no-trait controls isolating target-specific transfer. Our main finding is that larger independent datasets make the teacher's induced trait stand out more clearly in the student's later behavior. Other plausible traits may also strengthen with scale, but the target usually grows more. When the small-scale student already favors the target, scaling mainly amplifies that behavior; when it favors a related or salient alternative, more data can shift behavior toward the intended trait. Analyses of learned LoRA updates show a parallel trend. These effects appear across model families, trait types, multi-trait settings, and cross-model transfer. Our results suggest that scaling generated distillation data should be paired with trait-aware curation and evaluation, even when the data appears off-task or benign.
Zhichen Dong, Zhi-Xuan Liu, Yuanjiu Fan, et al.· 0 citations
Knowledge distillation transfers information from a high-capacity teacher to a smaller student, but its behavior across regression, classification, and heterogeneous tabular model families remains insufficiently understood. This paper presents a comparative simulation study of distillation in structured-data settings, where neural networks, tree ensembles, kernel methods, nearest-neighbor methods, and linear models are plausible competitors. Synthetic datasets with known ground truth and controlled noise are used to evaluate a three-model teacher ensemble distilled into multilayer perceptrons, tree ensembles, decision trees, kernel methods, instance-based methods, and linear models. Distillation weights and temperatures are selected on validation data, and test results are reported with confidence intervals, paired tests, and multiplicity-control checks. In regression, distillation acts as target smoothing and denoising: the teacher reduces label noise by 17.4%, 13 of 14 students improve numerically, and the MSE reduction is 7.1%. After multiplicity control, robust gains concentrate among MLPs. In classification, gains are smaller: 12 of 15 students improve, with an average accuracy gain of 0.5 percentage points, and soft-label distillation improves calibration for MLPs. Additional noise sweeps, teacher ablations, real-data checks, baseline comparisons, temperature analysis, and calibration results show that distillation helps most when the student has sufficient capacity, and the teacher provides a cleaner or more informative target. An extended set of confirmatory analyses—including teacher-quality controls with a shuffled negative control, ensemble-based uncertainty proxies, harder data conditions, distribution-shift robustness, a capacity ladder, computational-cost analysis, additional baselines and calibration metrics, an α–T factorial analysis, quantitative measures of soft-probability structure, and comparisons with modern tabular models—shows that useful transfer requires teacher predictions that remain conditionally aligned with the inputs. The classification gains are small in absolute accuracy but stable across seeds and are accompanied by improvements in selected calibration measures.
SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound on the distillation loss by applying the student's quantization parameters to quantize the teacher's features during distillation is proposed.
H. Lee, Hyeonsik Jo, Jinwook Chung et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
