Experimental results demonstrate that the ToolVerse framework significantly strengthens LLMs'capabilities in long-horizon tool use, achieving a marked performance boost and showcasing robust reasoning within dynamic environments.
Abstract
While LLM agents demonstrate strong reasoning abilities in compact and well-defined scenarios, they struggle to maintain robustness and effectiveness when faced with large-scale, diverse, and dynamic real-world environments that demand seamless tool integration. To address this gap, we introduce ToolVerse, a comprehensive framework that scales up agentic RL environments and enables agents to perform complex long-horizon reasoning in Tool-Integrated Reasoning (TIR) tasks. First, ToolVerse automatically builds the massive executable agent training environments from nearly 400 real-world Model Context Protocols (MCPs) that contain about 4500 tools. Second, we propose a task design strategy based on a tool dependency graph, utilizing Dynamic Unlocking Sampling Algorithm to generate long-horizon tasks, and produce GUST (Graph Unlocking Sampling Tasks) dataset. Third, to alleviate the credit assigment problem in long-horizon agentic RL, we propose a fine-grained Turn-Aware Relative Advantage algorithm. We conduct extensive Agentic RL training using ToolVerse and evaluate our framework on serveral agentic benchmarks. Experimental results demonstrate that our framework significantly strengthens LLMs'capabilities in long-horizon tool use, achieving a marked performance boost and showcasing robust reasoning within dynamic environments.
Experimental results on a broad range of MLE tasks with diverse model types and scales demonstrate that Matryoshka Agent is an effective and scalable paradigm for long-horizon MLE tasks and complex agentic problem solving.
Rushi Qiang, Changhao Li, Haotian Sun et al.· arXiv.org· 0 citations
SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments that combines a Gymnasium-compatible environment wrapper, a VERL-style rollout dataflow, group-relative policy optimization without a separate value model, and a sink-aware FlexAttention path designed to preserve model-specific sink scaling under causal and sliding-window masks is presented.
Zelei Cheng, Amritansh Mishra, Sambit Sahu et al.· 0 citations
ODYSSE is presented, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning, and an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO.
This work presents a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses, supporting unified training across heterogeneous execution systems.
Huatong Song, Fei Bai, Ming Yang et al.· 0 citations
Autonomous multi-modal agents are increasingly important in real-world applications due to their ability to reason about complex environments and orchestrate tool use. However, deploying multi-modal large language models (MLLMs) for tool use is often constrained by computational cost and inference latency, creating a pressing need for compact models that retain strong agentic capabilities. Training small multi-modal agents remains difficult: limited backbone capacity weakens multi-step reasoning, reward signals for tool use are often sparse and brittle, and naive distillation can fail to transfer the procedural knowledge required for reliable tool invocation and grounding. In this paper, we propose a two-stage self-evolutionary knowledge distillation framework that equips small MLLMs with robust and adaptive tool-use behaviors. Our method combines (i) mutual information-guided trajectory distillation, which selectively transfers high-utility segments of agentic trajectories from a larger teacher, and (ii) reinforcement-driven policy evolution with iterative teacher feedback. To stabilize learning and prevent semantic collapse, we introduce weighted semantic objectives and iteratively expand competence through error-driven optimization, hybrid experience replay, and group-relative policy refinement with multi-dimensional rewards over answer correctness, invocation validity, and tool effectiveness. Integrated with interactive tool modules, our approach enables small models to achieve strong performance across diverse tool-use benchmarks. Comprehensive experiments show consistent improvements over single-pass distillation and RL baselines. Overall, our framework provides a practical path to deploy efficient multi-modal agents without sacrificing tool-use reliability.
Lei Shen, Chengyu Wang, Yuanjie Lyu et al.· Proceedings of the 32nd ACM...· 0 citations
A one-round study provides initial evidence for PRD-guided self-evolution, motivating validation at larger scales and in industrial settings, and presents AgentOmnia, a framework coordinating task-space definition, data synthesis, post-training, evaluation, and improvement across To-Consumer (ToC), To-Business (ToB), and To-Employee (ToE) applications.
