Skip to content
Preprint

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Nov 2026 · 0 citations · 24 references
Computer Science

TL;DR

Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates, and accumulates context-relative returns in a global memory with balanced aggregation and reliability weighting to control bias and noise from heterogeneous tasks.

Abstract

Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.

View source

Similar papers

#natural language process... Preprint Oct 2026

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in envir...

Ru-Mei Li, Xun-Liang Cai · 0 citations
#artificial intelligence Preprint Oct 2026

Matched-Context Group Policy Optimization for LLM Agent Training

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context G...

Chen-Xu Zhang, Yao-Rui Shi, Wen-Long Zhang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization at the level of environment-facing segments that outperforms existing baselines on the ALFWorld and WebShop benchmarks.

Xin-Chen Du, Zheng-Ze Zhou, Wen-Hui Zhu et al. · 0 citations
Preprint Oct 2026

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a traj...

Zhengping Jiang, William Jurayj, Liaoyaqi Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents

Terminal feedback in multi-turn agent training leaves both rollout allocation and local credit underdetermined. Dynamics-Predictive Sampling (DPS) [9] provides a task-level predictive sampling reference, but interactive agents also need to choose which visited states merit further continuations. We introduce TSBPO with...

Rui Yu, Chong Mu, Jin-Chuan Zhang · 0 citations
Preprint Oct 2026

One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

This work proposes Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations, and shows that EMPO outperforms strong baselines in both task performance and learning efficien...

Shao-Jin Ma, Rui-Tao Wang, Peng-Yi Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.