Preprint

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

Tian Pan Yuan Li Hongda Wang Linbo Jin Meng Song Lei Gao Qiming Shi Shaokang Fu Jiarong Zhao Chengyu Wang Chengfu Huo

Aug 2026 · 0 citations · 32 references

Computer Science

TL;DR

This work proposes BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively, enabling agents to utilize external skills more effectively.

Abstract

External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts, and refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction.

Binbin Zheng, Zi-Jun Xie, Guan-Qun Zhao et al. · arXiv.org · 1 citation

Jul 2026

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model, is proposed.

Jinyang Wu, Shuo Yang, Zhengxi Lu et al. · arXiv.org · 7 citations

Jul 2026

Distilled Reinforcement Learning for LLM Post-training

Extensive experiments show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k, and can effectively transfer previously unavailable knowledge from a teacher model to a student model.

Chen Wang, Zhaochun Li, Jionghao Bai et al. · arXiv.org · 2 citations

Review Aug 2026

Agentic Reinforcement Learning with Self-Distilled Reward Shaping

ADRS is introduced, a framework for constructing return-associated token-level credit for multi-turn language agents that centers and normalizes privileged token scores within each step, modulates them with a return-associated Teacher Value Advantage gate based on within-group confidence--return association, and incorporates the gated token signal into native RL credit construction.

Ran Zhang, Guinan Chen, Chenshaodong et al. · 0 citations

Preprint Aug 2026

EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents

EviSD is proposed, an evidence-conditioned self-distillation framework that uses instance-level supporting evidence as privileged information for search actions and golden answers as complementary privilege for answer actions, and achieves the highest macro-average Exact Match in all evaluated settings.

Jianan Xie, Xin Sun, Zhongqi Chen et al. · 0 citations

Preprint Jul 2026

Cross-Benchmark Generalization in Long-Horizon Agents

Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.

Sushant Mehta, Logan Ritchie, Liudas Panavas et al. · 0 citations