Skip to content

TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs

Sep 2026 · arXiv.org · Vol abs/2609.33589 · 0 citations · 61 references
Computer Science

TL;DR

Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal, and reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget.

Abstract

Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as token-level credit using Jensen--Shannon (JS) divergence between the corresponding temperature-scaled next-token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11 benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.

View source

Similar papers

Conference Aug 2026

Post-Training for Reasoning LLMs with Reinforcement Learning: A Stability–Efficiency Perspective

Reinforcement Learning (RL)-based post-training has become a key mechanism for improving reasoning Large Language Models (LLMs), especially when automated verifiers enable Reinforcement Learning with Verifiable Reward (RLVR). Recent methods, however, are increasingly modular: many apparent algorithmic advances recombin...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 2 citations
#artificial intelligence Preprint Sep 2026

MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning

This work proposes a general RL-based framework for Distribution Matching allowing matching the distribution of a latent categorical attribute of model outputs to a specified target distribution and proposes reward functions for other divergences such as KL and Jensen-Shannon and motivate them with theoretical justific...

Sourabh Kulkarni, Ksheeraj Sai Vepuri, B. Demir et al. · 1 citation
Preprint Aug 2026

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

This work proposes two complementary strategies to improve the performance of value function RL: Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; and TETHER, a baseline that adaptively interpolates between gro...

S. Venkatraman, Matthieu Dinot, Laurence Aitchison · 1 citation · ⚡1

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

This work proposes self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit, and demonstrates that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Zhenrui Yue, Hui-Min Zeng, Yue-Qi Wang et al. · 1 citation
#small language model Preprint Aug 2026

Boosting LLM Exploration via Weak-Model Guidance in RLVR

This work empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training and efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.

Xin Shen, Huishuai Zhang, Peng Li et al. · 2 citations · ⚡1
#artificial intelligence Preprint Sep 2026

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization at the level of environment-facing segments that outperforms existing baselines on the ALFWorld and WebShop benchmarks.

Xin-Chen Du, Zheng-Ze Zhou, Wen-Hui Zhu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.