Graft-Oriented Distillation (GOD) is proposed, a component-level distillation framework for improved generalization through grafting, which uses selected frozen-teacher components with trainable student counterparts to build hybrid source models.
Abstract
Sequential recommenders often struggle with sparse and noisy histories, limiting generalization to unseen interactions. Knowledge distillation mitigates this by transferring dense supervision from a teacher to a student. However, most distillation methods run teacher and student independently, then match student outputs or representations to the teacher. Such supervision entangles student-component effects, blurring whether weak generalization stems from unreliable embeddings, overfitted encoding, or co-adaptation to sparse histories. In this paper, we propose Graft-Oriented Distillation (GOD), a component-level distillation framework for improved generalization through grafting. Grafting denotes replacing selected frozen-teacher components with trainable student counterparts to build hybrid source models. GOD uses these hybrid models to evaluate student embeddings with the teacher encoder and the student encoder with teacher embeddings, providing component-level feedback. At inference, GOD uses only the student, incurring no additional cost. Across three real-world datasets, GOD outperforms state-of-the-art baselines by up to 13.92%.
SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound on the distillation loss by applying the student's quantization parameters to quantize the teacher's features during distillation is proposed.
H. Lee, Hyeonsik Jo, Jinwook Chung et al.· 0 citations
Privileged Self-Distillation (PSD) is proposed, a framework that separates learning-time information from inference-time information and uses an advantage-reachability gate to focus distillation on teacher signals likely supported by the observed prefix, along with a momentum-averaged teacher for stable targets.
Jiakai Tang, Yang Zhang, See-Kiong Ng et al.· arXiv.org· 1 citation
This work reformulates continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits and shows that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures.
This study introduces a novel memory-augmented self-learning framework that extracts and provides diverse learning sources for adaptive knowledge distillation from the student model itself, resulting in a 2.5-6% increase in accuracy across various benchmark datasets compared to current GNN training and self-distillation methods.
Saurabh Sharma, Souvik Chowdhury, Joydeep Chandra· Data mining and knowledge di...· 0 citations
Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.
Francisco Galuppo Azevedo, Clarissa Lima Loures· 0 citations
Knowledge distillation (KD) has become a pivotal technique for transferring knowledge from large-scale teacher models to lightweight student models. However, traditional feature-based distillation methods necessitate the direct exposure of the teacher’s intermediate representations, raising concerns regarding data privacy and the leakage of proprietary model details. These concerns often hinder the deployment of distillation in collaborative or cloud-based scenarios. To address these challenges, we propose a privacy-aware teacher-oriented projected feature distillation framework that reduces direct feature exposure while preserving distillation effectiveness. Specifically, our method employs a low-rank projection strategy to obfuscate the teacher’s features into a compact subspace. Authorized student models leverage this projection matrix to align their own features, enabling effective knowledge transfer while reducing direct exposure of the teacher’s original spatial patterns. Extensive experiments on benchmark datasets demonstrate empirical feature obfuscation and reconstruction resistance while maintaining competitive performance in object detection and semantic segmentation, with gains in several settings even when learning from projected representations.
Jun-Fei Yi, Sihao Lin, Hui Zhang et al.· IEEE Transactions on Image P...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
