Skip to content

Beam Search as Test-Time Self-Distillation via Counterfactual Contexts

Sep 2026 · arXiv.org · Vol abs/2609.37041 · 0 citations · 34 references
Computer Science

TL;DR

Test-time self-distillation is proposed, a decoding-time method that extracts a steering signal from the self-distillation framework without any parameter updates, reward models, or training data, demonstrating that the self-distillation principle can be operationalized at inference time.

Abstract

Self-Distillation Fine-Tuning (SDFT) enables a language model to act as its own teacher: by conditioning on a demonstration, the model produces an implicit reward via pointwise mutual information, which guides on-policy learning without external supervision. However, SDFT operates at training time: it requires gradient updates and access to expert demonstrations, making it inapplicable at inference. We propose test-time self-distillation, a decoding-time method that extracts a steering signal from the self-distillation framework without any parameter updates, reward models, or training data. Our key insight is that counterfactual contexts, i.e. fixed textual templates that hypothetically prime the model for excellent versus poor reasoning, can substitute for the demonstration. The log-odds ratio of a candidate answer under these two counterfactual conditions defines a new reward signal. We derive the optimal KL-regularized policy under this reward, which takes the form of a Gibbs reweighting of the base distribution. Crucially, this reweighting is global: it cannot be decomposed into independent per-token operations without ignoring future trajectory quality. We therefore approximate the target distribution via beam search. Experiments on mathematical reasoning (MATH500), code generation (HumanEval), and graduate-level science QA (GPQA) across multiple model scales show that test-time self-distillation improves over standard sampling, low temperature, beam search and power sampling baselines on average, demonstrating that the self-distillation principle can be operationalized at inference time.

View source

Similar papers

#artificial intelligence Preprint Nov 2026

Speculative Self-Distillation enables Efficient Knowledge Internalization

Many real-world applications require language models to internalize knowledge that was absent from pretraining, to incorporate continually updated facts, or to learn new domain skills. Self-distillation has emerged as a natural approach for this task. A model conditioned on the source material acts as the teacher, and...

Shayan Talaei, A. Bhatia, Arshia Soltani Moakhar et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Does self-distillation improve reasoning?

This paper revisits two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), and finds that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than eff...

J. Adhinarta, Raina Wu, Ashutosh Sharma et al. · 0 citations
#natural language process... Preprint Oct 2026

Self-Distillation with Verbal Regularization

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distribut...

Jiazheng Li, Yawei Li, Jun-Ru Lu et al. · 0 citations

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

Controling how much and when teacher information is introduced improves specialisation while preserving broader capability is shown, and results show that controlling how much and when teacher information is introduced improves specialisation while preserving broader capability.

Ahmed Khaled Khamis, Xiao-Tong Ji, Hassan Jaber et al. · 1 citation

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation
#machine learning Preprint Oct 2026

One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

On-policy self-distillation (OPSD) improves reasoning by training a model on its own trajectories with dense token-level supervision from a privileged copy conditioned on a reference solution. Existing methods typically use a single teacher distribution conditioned on the complete reference. We identify a limitation of...

Khouloud Saadi, Di Wang · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.