This work frames this as a natural-language prediction problem over the PerturbQA benchmark and train a calibrated predictor via reinforcement learning on a distillation-initialized 8B-parameter language model and implements calibration-guided test-time compute that selectively allocates extra inference budget to low-confidence predictions using budget forcing and self-consistency.
Diagnostics reveal that RL on PLMs is governed by two reward properties: verifiability, whether the reward is a fixed environment or a learned surrogate vulnerable to distribution shift, and coverage, the fraction of sequence space giving an informative gradient.
Hanqun Cao, Hongrui Zhang, Junde Xu et al.· Proceedings of the 32nd ACM...· 0 citations
Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.
Sushant Mehta, Logan Ritchie, Liudas Panavas et al.· 0 citations
TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations, is proposed and it is found that learnability is reproducible across independently sampled training contexts and predictive of downstream utility.
Ting Zhou, Zhenqing Ling, Daoyuan Chen et al.· 0 citations
A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.
Martino M. L. Pulici, Cuong Xuan Chu, Evgeny Kharlamov et al.· arXiv.org· 0 citations
GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, is proposed, and is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures.
Xiaodong Liu, Michael Xu, Jack W. Stokes et al.· arXiv.org· 0 citations
Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.
Le-Qi Zheng, Jin-Bo Su, Fang Niu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
