Skip to content

ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

Sep 2026 · arXiv.org · Vol abs/2609.35433 · 0 citations · 25 references
Computer Science

TL;DR

This work proposes ReSPO (Reshaped Sequence Policy Optimization), a smooth, two-branch sequence-level kernel derived from an $\alpha$-divergence variational objective and an exponential variance-control tilt that effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail.

Abstract

Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an $\alpha$-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.

View source

Similar papers

#machine learning Preprint Oct 2026

Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization

Off-policy data in reinforcement learning with verifiable rewards (RLVR) can arise from asynchronous generation, rollout-training mismatch, and response reuse. Token-wise importance weighting avoids multiplying likelihood ratios across a response but generally biases policy gradients. We prove that, under coverage and...

Hao Bai, Jiarui Yao, Tong Zhang · 0 citations

EasyPPO: Stabilizing the Critic Is Key

A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for la...

Xuan-Yi Zhou, Qiu-Yang Mang, Huan-Zhi Mao et al. · 0 citations
#machine learning Preprint Oct 2026

Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which...

Xuan-Chang Zhang, Tong Zhang · 0 citations
#artificial intelligence Preprint Oct 2026

TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning

This paper characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify and introduces TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively...

Zhen Li, Shuai Zhang, Yang-Gan Gu et al. · 1 citation
#artificial intelligence Preprint Oct 2026

FERPO: Forward Entropy-Regularized Policy Optimization

Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unr...

Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.