NLPG is proposed, an external policy-memory method for improving a fixed agent without changing its model parameters or program structure that outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average.
Abstract
Large language model agents increasingly rely on compound programs for retrieval, tool use, reasoning, and verification, yet their failures often arise from local procedural decisions. Existing reinforcement-learning and prompt-optimization approaches typically rely on scalar rewards or repeatedly modify entire prompts, making it difficult to capture and reuse procedural improvements while preserving a frozen agent. To address this problem, We propose Natural-Language Policy Gradients (NLPG), an external policy-memory method for improving a fixed agent without changing its model parameters or program structure. NLPG diagnoses execution traces, propagates downstream feedback backward through the module graph, and converts recurring failures into route-local natural-language corrections that are aggregated into bounded policy updates for subsequent executions. Across six benchmarks covering memory, reasoning, instruction following, and evidence verification, NLPG also outperforms the strongest listed baseline for each benchmark by 8.71 percentage points on average. These results provide evidence that evaluated procedural experience can be transformed into local and interpretable policy updates, enabling continual improvement of frozen agents.
This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...
Liu Yang, Han Zhu, Zheng-Yang Zhong et al.· Unmanned Systems· 1 citation
Textual skills enable large language model (LLM) based agents to accumulate reusable procedural knowledge without updating model parameters. Yet existing skill evolution remains largely confined to the text space: an optimizer must diagnose success and failure patterns, and revise skills solely from long execution traj...
Kai-Xin Zhang, Chang-Ming Li, Ying-Dong Shi et al.· arXiv.org· 0 citations
Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.
Wen-Bo Gao, Zhao-Mou Song, Zhi-Yuan Ji et al.· arXiv.org· 1 citation
This paper proposes FRESH, a Failure-aware Retrieval framework over Experience-Structured Heterogeneous graphs, which transforms historical successes and failures into structured external experience for tool-using agents and consistently improves task success and tool-use reliability over no-memory agents and represent...
Jia-Xing Li, Lei Song, Rui Dong et al.· arXiv.org· 0 citations
A dense, decision-level reward is introduced in which an LLM judge evaluates the necessity of each tool call, which effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.
Yi-Wei Yang, Haoxiang Zhang, Bing-Bing Wen et al.· 0 citations
RAISE is introduced, which trains policy synthesizers from formal verification in two stages, verified supervised fine-tuning (SFT) followed by a reinforcement learning (RL) stage that learns from verifier signal, and it is found that SFT succeeds largely by letting models express authorization logic they already have.
Ying-Ming Zhou, Adarsh Vatsa, William Eiers· 0 citations