Skip to content

Self-Play Search Distillation for Large Language Model Reasoning

Sep 2026 · arXiv.org · Vol abs/2609.30936 · 0 citations · 59 references
Computer Science

TL;DR

Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games, offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

Abstract

Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

View source

Similar papers

OPSRD: On-Policy Self-Role Distillation

OPSRD is introduced, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions for on-policy self-distillation without reference solutions.

Wei-Jie Ren, Yan-Wen Zhang, Hao Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models

SOLID is proposed, a novel framework for self-improving OR language models without verified answers or external evaluators that improves solution accuracy for both general-purpose and OR-tuned models over outcome-only group-relative training.

Rui-Chen Zhu, Ming-Long Cao, Chen-Yu Zhou et al. · 0 citations
Preprint Aug 2026

Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills

This work shows that reasoning traces are not a prerequisite: skills distilled from non-reasoning trajectories alone remain competitive with skills distilled from paired reasoning/non-reasoning corpora, with domain-dependent differences between the two sources.

Agamdeep Singh, Srishti Gautam, Priyanshu Gupta et al. · 0 citations
#machine learning Preprint Sep 2026

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Negative Self-Distillation (NSD) is introduced, a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions, and consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.

Rong-Can Pei, Zhepei Wei, Shu-Yao Xu et al. · 1 citation

Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces

Evaluated on two reasoning domains, math and multi-hop reasoning, across 1-8B models from different families, LoT consistently improves over KD and converges faster than staged curricula, highlighting the value of adaptive progression.

Ming-Hui Liu, Thomas Magelinski, De-Hao Yuan et al. · 0 citations

Learning from Think-Mode Advantage via On-Policy Distillation

ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.

Wan-Qi Ren, Jian-Xiang Wang, Dan-Xuan Liu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.