H2SD: Hybrid Hindsight Self-Distillation

Qi Cai Yi-Chuan Ma Linyang Li Pei-Ji Li Yong-Kang Chen Qipeng Guo Yicheng Zou Tao Gui Xiaocheng Feng Bing Qin

Jul 2026 · arXiv.org · Vol abs/2607.18955 · 2 citations

Computer Science

TL;DR

Experiments on challenging reasoning benchmarks show that H$^2$SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.

Abstract

Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but a scalar trajectory reward offers limited token-level guidance. Existing self-distillation methods add a privileged teacher but typically assign it a fixed role: direct distribution matching may destabilize successful behavior, while magnitude-only modulation offers little corrective guidance after failure. We observe that successful and failed trajectories require different forms of hindsight supervision. A successful response already contains a valid student-generated reasoning path and can therefore serve as privileged context rather than being replaced by an external rationale. A failed response, however, requires corrective reference information. We introduce Hybrid Hindsight Self-Distillation ($\mathrm{H}^{2}\mathrm{SD}$), which jointly adapts teacher context and update strategy to trajectory correctness. For successful trajectories, we construct the teacher context from the verified response and a rephrasing instruction, and use the teacher only to re-evaluate the original response tokens. The resulting probabilities refine token credit assignment without changing the direction determined by the reward. For failed trajectories, a verified reference hint provides corrective guidance through reverse-KL distillation. Experiments on challenging reasoning benchmarks show that H$^2$SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts, and refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction.

Binbin Zheng, Zi-Jun Xie, Guan-Qun Zhao et al. · arXiv.org · 1 citation

Preprint Aug 2026

EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents

EviSD is proposed, an evidence-conditioned self-distillation framework that uses instance-level supporting evidence as privileged information for search actions and golden answers as complementary privilege for answer actions, and achieves the highest macro-average Exact Match in all evaluated settings.

Jianan Xie, Xin Sun, Zhongqi Chen et al. · 0 citations

Preprint Aug 2026

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

This work proposes AHEAD, a step-aware framework that matches different supervision sources to different step types and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.

Xiaolong Jin, Dingmin Wang, Vijay Lingam et al. · 0 citations

Preprint Aug 2026

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

This work reproduces SDPO's reported gains in its easy setting, then applies the identical setup to difficult tasks and finds that it does not teach anything, and explains this failure through a single causal chain from the loss to the model it produces.

Sarthak Harne, Chinmay Karkar, Yash Pandya et al. · 3 citations

Preprint Aug 2026

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent and uses imitation only where group-relative rewards are uninformative, obtains the best result in all four scientific domains.

Yubo Zhang, Xin-Hong Ma, Zezhong Tan et al. · 0 citations

#machine learning Preprint Sep 2026

Cliff: Learning Process Rewards from the First Mistake

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth''in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Peixuan Han, Runnan Wang, Ketan Ramaneti et al. · 1 citation