Harness-G, a graph-structured retrieval framework that reformulates free-form query generation as finite action selection, and introduces Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them.
Abstract
Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.
A small language model is trained via supervised fine-tuning followed by reinforcement learning to jointly perform agent selection and structured parameter generation for downstream tool calls, using a hierarchical reward function grounded in retrieval relevance along with query-agent topic alignment to learn task-dependent agent suitability from retrieval performance.
Gayathri V Kondapalli, Alexander Ng, Hirsh Pithadia et al.· 0 citations
Guided Retrieval Training (GRT) is introduced, a novel method that improves the performance of a search agent by restricting the retrieval process during RL training using ground truth information, and enhances training efficiency by achieving better QA performance with fewer training steps.
Aounon Kumar, Sudipta Paul, Vivek Kulkarni et al.· 0 citations
Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leverages LLMs to improve retrieval through query expansion, data synthesis, and retrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduce CoGR, a retrieval framework that instead trains LLMs to directly construct retrieval representations on both query and item sides. Each generator produces a compact set of keywords, which are matched directly through an inverted index, preserving compatibility with existing keyword-based retrieval infrastructure. CoGR uses a two-stage training pipeline. Supervised fine-tuning first establishes an aligned keyword space, after which co-evolving reinforcement learning alternately optimizes the query- and item-side generators with GRPO against the opposite side's frozen index. Both sides optimize the same query-to-item retrieval $F_1$ objective: the query side receives retrieval $F_1$ directly, while the item side receives a counterfactual marginal reward measuring the change in query-side $F_1$ caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines, CoGR achieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving $F_1$ over the strongest baseline by $10.9\%$ and $36.1\%$, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.
Runpeng Dai, Kai-Li Huang, Changsung Kang et al.· 1 citation
Retrieval-Augmented Generation (RAG) has become a fundamental paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, while recent structure-augmented approaches organize documents into graphs to improve information access, their retrieval strategies remain largely static, relying on similarity ranking or static probability diffusion. We identify that this paradigm suffers from two inherent limitations in complex reasoning: popularity bias, where retrieval paths are trapped by high-degree distractors, and signal decay, where relevance signals attenuate over long reasoning chains. To overcome these challenges, we propose NaviRAG, a novel framework that reformulates retrieval as a reinforcement learning-driven dynamic navigation problem on schema-less knowledge graphs (KGs). Unlike passive diffusion, NaviRAG employs an agent that actively traverses the graph to act as a search-space pruning engine, identifying logical multi-hop reasoning paths. Technically, we introduce three key components: (1) Structure-Aware Query Expansion, which bridges the modality gap between unstructured queries and structured graph seeds for precise initialization; (2) Target-Driven Reward Shaping, which provides dense supervision based on semantic progress toward gold documents, effectively mitigating the sparse reward problem in large-scale graph traversal; and (3) a Multi-View Hybrid Reranking strategy that operates on the highly-pruned candidate subgraph, integrating policy confidence, semantic relevance, and global structural importance to ensure robust candidate selection. Extensive experiments on three multi-hop QA datasets and two single-hop QA datasets demonstrate that NaviRAG significantly outperforms baselines, achieving state-of-the-art performance in multi-hop QA while maintaining robustness in single-hop QA. Our code and data are available at https://github.com/CkingEW/NaviRAG.
Jinghong Lei, Wang Kun, Zhigang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Search-on-Graph-R1 (\sogrone{}), which internalizes this navigation into a compact 8B model through supervised fine-tuning (SFT) followed by reinforcement learning (RL), which surpasses every frozen frontier-LLM system in their comparison and posts the strongest results on CWQ of any system the authors compare against.
J. Sun, Hao Yu, Fengran Mo et al.· arXiv.org· 1 citation· ⚡1
This work presents \textsc{GTA-RAG}, a graph-trajectory-augmented RL framework for multi-turn retrieval-augmented reasoning that consistently outperforms RL-based RAG baselines with both Qwen2.5-3B and Qwen2.5-7B backbones, while substantially improving evidence-chain coverage.
Jun Chen, Yongchao Liu, Pengyu Qiu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
