Experience Augmented Policy Optimization for LLM Reasoning

Jinda Lu Kexin Huang Junkang Wu Shuo Yang Jinghan Li Chiyu Ma Shaohang Wei Xiang Wang Guoyin Wang Jingren Zhou

Jun 2026 · arXiv.org · Vol abs/2606.30420 · 2 citations · 41 references

Computer Science

TL;DR

This work proposes Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout to ensure stable and unbiased learning from experience-augmented rollouts.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

This work proposes ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization, enabling sustained performance improvements over extended training horizons.

Kexin Huang, Junkang Wu, Jinda Lu et al. · arXiv.org · 0 citations

Jul 2026

PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

This work proposes a policy-centric training paradigm that reframes skills as a dynamic training scaffold and converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts.

Yipeng Shi, Zhi-Peng Ma, Yue Wang et al. · arXiv.org · 0 citations

Preprint Jul 2026

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Single-rollout Asynchronous Optimization (SAO) is presented to address the stability and off-policy challenges in asynchronous RL and is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks.

Zhenyu Hou, Yujiang Li, Jie Tang et al. · 10 citations · ⚡2

Jul 2026

ISO: An RLVR-Native Optimization Stack

Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through t...

Hanqing Zhu, Wenyan Cong, Zhizhou Sha et al. · arXiv.org · 0 citations

Jul 2026

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

VarIance Guided Online Rollout allocation (VIGOR) is proposed which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budge...

Heyang Jiang, Henry Liu, Baharan Mirzasoleiman · arXiv.org · 1 citation

Jul 2026

TAPO: Transition-Aware Policy Optimization for LLM Agents

This work proposes TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision, and demonstrates that TAPO consistently improves task performance over pure policy optimization baselines.

Cong Li, Pei-Xi Peng, Yisen Zhao et al. · arXiv.org · 1 citation