Skip to content

Distilling Directional Verification

Oct 2026 · 0 citations · 38 references
Computer Science

TL;DR

Directional verification mitigates the transfer of errors from teacher-generated answers to students by providing more accurate training targets, and introduces directional label distillation, in which frozen teachers score candidate answers in that known direction and the best-scoring candidate becomes the student's training target.

Abstract

Knowledge distillation aims to transfer the factual knowledge of large language models to smaller models for efficient deployment. Yet a teacher may recall a relation in one direction while failing to generate the answer in the reverse direction. Distillation from its generated answers can therefore propagate this directional limitation to the student. The same teacher can nevertheless recognize such an answer by scoring the relation in the direction it knows. We introduce directional label distillation, in which frozen teachers score candidate answers in that known direction and the best-scoring candidate becomes the student's training target. On facts about parents and their children, known-direction scoring yields more accurate labels than scoring the requested direction, even after tuned corrections for name priors. With prior-corrected scores, the better direction depends on the facts rather than the template, and reverses on mined facts whose notable entity is the parent rather than the child. With the evaluated children's forward facts withheld, students trained on known-direction labels improve open-ended accuracy on their trained queries by 13 to 15 points over students trained on prior-corrected reverse labels. After generated answers are matched to a fixed name list by lexical similarity, students reproduce nearly all selected labels. Their accuracy largely follows label quality. The label advantage holds on unscreened queries and when candidates are retrieved without inserting correct answers. Our findings show that directional verification mitigates the transfer of errors from teacher-generated answers to students by providing more accurate training targets. Code is available at https://github.com/js-lee-AI/directional-verification.

View source

Similar papers

#machine learning Preprint Sep 2026

K2P: Label-Free Knowledge to Prompt Distillation

Knowledge distillation can transfer reasoning from stronger teachers to frozen students through reusable prompts, but avoiding weight updates does not eliminate supervision. Without ground-truth answers, teacher solutions are unverified, and agreement with the teacher can reward shared mistakes. We introduce Knowledge-...

Ying-Chuan Zhang, Hao-Ran Lu, Wen-Xuan Zhong et al. · 0 citations
#natural language process... Preprint Oct 2026

Correctness Is a Direction: Geometric Answer Selection in Language Models

Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models. We show that the mean displacement from incorrect to correct answer representations, computed at approximately 70\% of model depth from fifty labeled examples with no parameter updates, yields a scoring direction...

M. Armstrong, Navid Ayoobi, Pradham Mummaleti et al. · 0 citations

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

CaRE-KD is proposed, a confidence-gated distillation framework that replaces static objectives with uncertainty-adaptive optimization and provides a gradient-level analysis showing how this dual-granularity design induces a conditional calibration mechanism that prior static divergences cannot reproduce.

Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty · 0 citations
#artificial intelligence Preprint Sep 2026

Probing Factual Knowledge Transfer with Training Data Interventions

The results show that fact transfer is very limited: under the strictest removal condition, a large majority of English-acquired facts fail to transfer into Persian, and it is shown that sentence-level co-occurrence removal is insufficient to eliminate fact signal.

Romina Oji, Marc Braun, Marcel Bollmann et al. · 0 citations
#natural language process... Preprint Sep 2026

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Switch Distillation is proposed, a simple mid-training objective that distills on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal, and otherwise falls back to cross-entropy, which consistently outperforms existing distillation objectives across teacher sizes.

Jacqueline He, Howard Yen, S. Li et al. · 0 citations

Inductive Feedback for Mixed-Policy Distillation

This work treats verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix, and adopts a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback.

Amir Moeini, Huai-Jiang Zhu, Daniel Havir et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.