This hands-on problem-solving tutorial provides both a rigorous algorithmic and practical introduction to multi-turn RL finetuning for LLMs, and covers state-of-the-art multi-turn RL finetuning algorithms, turn-level vs. trajectory-level reward design, and production grade monitoring for reward hacking detection.
Zhe Wang, Sapana Chaudhary, Jiayu Li et al.· Proceedings of the 32nd ACM...· 0 citations
Production large language model (LLM) based systems such as coding agents, web navigators, and tool-calling assistants operate over multiple turns of interaction with users, tools, and environments. Pretrained LLMs, depending on their size, can either underperform in these settings due to misalignment with the system's interaction mechanics, or, when capable, incur prohibitive latency. Fine-tuning right-sized models addresses both accuracy and latency, but training such multi-turn agents requires Reinforcement Learning (RL), where the model acts as a policy optimizing long-horizon outcomes across sequential interactions. This poses challenges absent from single-turn settings: credit assignment over long trajectories, reward design for sparse and delayed feedback, state and context management as observation histories grow, environment scaling for parallel rollout collection, and training stability under prompt/environment distribution shift. This hands-on problem-solving tutorial provides both a rigorous algorithmic and practical introduction to multi-turn RL finetuning for LLMs. Using Amazon SageMaker AI, participants progress through four labs: (1) environment and reward function design, (2) multi-turn trajectory collection and Group Relative Policy Optimization (GRPO)-based training, (3) reward densification and credit assignment strategies, and (4) evaluation, failure diagnosis and deployment. We cover state-of-the-art multi-turn RL finetuning algorithms, turn-level vs. trajectory-level reward design, and production grade monitoring for reward hacking detection. The tutorial targets machine learning (ML) engineers, data scientists, and researchers who build agentic LLM systems. No prior RL experience is required. All materials will be publicly available on GitHub.
Zhe Wang, Sapana Chaudhary, Jiayu Li et al.· Proceedings of the 32nd ACM...· 0 citations
This work introduces AutoWorldModel-Bench, a closed-loop benchmark in which frontier coding agents autonomously improve a provided base world model under a fixed compute budget, and offers a setting in which frontier coding agents can be evaluated on open-ended research rather than engineering-to-spec problems.
Marjan Moodi, Xuan Zhu, F. Silva et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
