Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization at the level of environment-facing segments that outperforms existing baselines on the ALFWorld and WebShop benchmarks.
Abstract
Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error can cause task failure, while trajectory-level rewards provide limited guidance for assigning credit to individual decisions. To address this limitation, we introduce Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization (GRPO) at the level of environment-facing segments. Inspired by the existing on-policy self-distillation (OPSD) method, SHARPO computes teacher-student log-probability gaps within each segment and uses the resulting signal to compute a bounded multiplier on the GRPO advantage. This multiplier is shared by all tokens within the segment, allowing credit to vary across different segments. With Qwen2.5-7B-Instruct, SHARPO outperforms existing baselines on the ALFWorld and WebShop benchmarks, including GRPO, SDAR, RLSD, and StepOPSD.
LLM Value Function RL is introduced, which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic, and consistently and substantially outperforms both GRPO and PPO.
Kai-Jie Zhu, Wei-Xiang Yan, Omar Attia et al.· 1 citation
It is argued that the right resolution is state-dependent, and GACA, a critic-free estimator whose granularity follows an uncertainty-based criticality proxy is proposed, improves task success over GRPO and GiGPO at both 1.5B and 7B scales.
Tao Liang, Yang Liu, Shang Luo et al.· 3 citations
Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task re...
Jin-Hao Dong, Liang Zhao, Zi-Hao Yue et al.· 1 citation
Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates, and accumulates context-relative returns in a global memory with balanced aggregation and reliability weighting to control bias and noise from heterogeneous tasks.
Xuan-Ren Chen, Chong-Yang Tao, Wen-Peng Hu et al.· 0 citations
AdaStep is proposed, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal, and an optimal per-state shrinkage coefficient is derived.
X. Wang, Wen-Hao Wu, Meng-Hao Zhang et al.· 0 citations
This work proposes self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit, and demonstrates that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.
Zhenrui Yue, Hui-Min Zeng, Yue-Qi Wang et al.· arXiv.org· 1 citation
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.