Skip to content
Preprint

Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift

Sep 2026 · 1 citation · 45 references
Computer Science

TL;DR

Rita is proposed as a novel RL paradigm that performs consistently superior to the supervised finetuning approaches and vanilla RL-finetuned frameworks and adopts a difficulty-aware data filtering strategy that selects informative easy-to-medium samples for RL using rollout error rate and reward variance.

Abstract

This paper studies \textbf{thinking--answer consistency} in vision-language models. We focus on Visual Intention Grounding, where a model infers a target object based on a human intention query and predicts a bounding box. We reveal that previous IoU-based reinforcement learning (RL) frameworks suffer from ``thinking drift'', where the model produces a correct bounding box, despite having an incorrect reasoning process pointing to a different target object. Thus, we propose \textbf{Rita} (\textit{ReInforcing Thinking--Answer consistency}) as a novel RL paradigm to tame the drift. Specifically, Rita introduces two reasoning-label-free RL rewards, constructed from the conditional probability of reference answers: a \textbf{thinking reward} and a \textbf{consistency reward}. It also adopts a difficulty-aware \textbf{data filtering} strategy that selects informative easy-to-medium samples for RL using rollout error rate and reward variance. Extensive experiments on EgoIntention and the new RefEgo-Int benchmarks show that Rita performs consistently superior to the supervised finetuning approaches and vanilla RL-finetuned frameworks.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

To Think or Not to Think: Allocating Reasoning Where It Helps

This paper proposes CARE, a method which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost.

Zheng-Dong He, Yun-Fan Zhou, Jian-Guo Yao et al. · 0 citations
#natural language process... Preprint Sep 2026

MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves re...

Zineddine Tighidet, A. Mogini, Jia-Li Mei et al. · 0 citations
#natural language process... Preprint Oct 2026

The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models

Reasoning models allocate extra computation at inference time and present their answers as the product of deliberate thought. If this deliberation works the way dual-process accounts of human cognition suggest, longer thinking should weaken the classic decision biases that fast, intuitive judgment produces. Using 30 vi...

Obada Kraishan · 0 citations
#machine learning Preprint Sep 2026

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

This work operationalizes the importance of a reasoning step as its advantage: the change in expected reward, e.g., producing the correct final answer, from including that step, estimated via Monte Carlo rollouts, and evaluates whether LLM judges can identify high-advantage steps and finds that sufficiently capable LLM...

Kevin Du, A. Hoyle, L. Ruis et al. · 1 citation
#natural language process... Preprint Oct 2026

Thinking Inertia: LLMs Keep Thinking When Told Not To

Large Language Models (LLMs) increasingly ship with explicit"thinking modes", yet their counterpart,"no-thinking", has received far less attention. We study LLMs'no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode...

Dian-Qiao Lei, K. Lin, Pan Lu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Thinking effort aligns between humans and reasoning models in abductive reasoning

This work isolates abductive reasoning from deductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort.

Henry Arthur · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.