Measuring Reward-Seeking via Contrastive Belief Updates

Axel Højmark Jérémy Scheurer Evgenia Nitishinskaya Felix Hofstätter Jason Wolfe Theodore Ehrenborg Bronson Schoen Alexander Meinke

Jul 2026 · arXiv.org · Vol abs/2607.18966 · 2 citations

Computer Science

TL;DR

The results indicate that RL can increase reward-seeking over the course of training, producing models that may act against their developers'intentions when they believe that doing so leads to higher reward.

Abstract

Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective. This"reward-seeking"is difficult to measure because a model that pursues the grader's judgment and one that pursues the intended objective behave identically whenever the grader rewards the intended behavior. We measure reward-seeking using Contrastive Synthetic Document Finetuning to change a model's beliefs about what the grader rewards, putting those beliefs in conflict with what users or developers want, and measuring the rate at which the model adopts each party's preferred behavior. Applied to intermediate checkpoints of a capabilities-focused OpenAI o3 RL run, without safety training, we find that these checkpoints often side with grader preferences over those of users or developers on coding and alignment tasks. This tendency to side with the grader trends upward throughout RL training. For example, in an environment that forces a choice between keeping a promise to a supervisor and breaking it to complete the task, a late capabilities-focused o3 checkpoint breaks the promise 87% of the time when SDF documents say the grader rewards task completion, versus 9% when they say it rewards honesty (a choice its chain-of-thought often makes explicit). An earlier checkpoint is far less sensitive (40% vs. 24%). Our method also generalizes to reward-hacking models. A model organism trained to reward-hack (gpt-oss-120b) is more than twice as sensitive to grader preferences as the unmodified model, with the mean behavioral shift in favor of the grader rising from 33% to 86%. These results indicate that RL can increase reward-seeking over the course of training, producing models that may act against their developers'intentions when they believe that doing so leads to higher reward.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

The proposed SeekJudge framework, in which four role-specialized agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek--Analyze loop over the trajectory, is the first practical model-based reward to match or surpass native rule-based supervision in online RL.

Yang Wan, Zhenhao Zhang, Jie-Rui Wang et al. · arXiv.org · 0 citations

#machine learning Preprint Sep 2026

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.

Le-Qi Zheng, Jin-Bo Su, Fang Niu et al. · 2 citations

Preprint Jul 2026

Cross-Benchmark Generalization in Long-Horizon Agents

Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.

Sushant Mehta, Logan Ritchie, Liudas Panavas et al. · 0 citations

Jul 2026

Rewarding Better Thinking for LLM Preference Alignment

Think Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment that converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations.

Xu-Bo Liu, Wenya Guo, Ruxue Yan et al. · arXiv.org · 0 citations

Jul 2026

AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning

AdaKP is an online selector that re-chooses each problem's KP subset over the course of RL training, an entropy proxy that scores a KP by the reduction in next-token entropy it induces in a single inexpensive forward pass, with a provable bound on its truncation bias.

Zibin Meng, Zhenyu Zhao, Chunqiang Run · arXiv.org · 0 citations

#artificial intelligence Review Aug 2026

A Survey on Rubric-Guided Reinforcement Learning for Language Models

A Bayesian framework that defines constitutions as prior distributions over evaluation criteria and rubrics as conditional instantiations is introduced, and a taxonomy of rubric-guided RL along the prior-posterior axis is presented, covering constitutional AI, instance-specific rubrics, process-level supervision, self-evolving rubrics, and their agentic and multimodal extensions.

Zifei Shan, Fang-Ning Shao · 0 citations