Skip to content

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

Sep 2026 · arXiv.org · Vol abs/2609.35954 · 0 citations · 35 references
Computer Science

TL;DR

This work introduces ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations.

Abstract

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Function-Space Temporal Self-Distillation for Policy Optimization

This work introduces Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation, and introduces privileged disagreement correction to explicitly suppress locally overvalued actions.

Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma · 0 citations

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout effici...

Yu-Xiao Yang, Shang-Zhe Li, Tian-Run Yu et al. · 0 citations
#natural language process... Preprint Oct 2026

OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents

Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outc...

Yu-Jia Luo, Zhi-Yi Lai, Zi-Qi Miao et al. · 0 citations

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

This work proposes self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit, and demonstrates that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Zhenrui Yue, Hui-Min Zeng, Yue-Qi Wang et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.