Mitigating Think-Answer Mismatch in large language model reasoning through noise-aware advantage reweighting
Abstract
Group-Relative Policy Optimization (GRPO) is a key technique for training large reasoning models, but its reliance on outcome rewards makes it vulnerable to Think-Answer Mismatch, where final-answer correctness does not faithfully reflect reasoning quality. This vulnerability is amplified in highly imbalanced response groups, where a single corrupted reward can strongly distort group-relative advantages. To address this problem, we propose Stable Group-Relative Policy Optimization (S-GRPO), which derives a closed-form, noise-aware weight by minimizing the expected squared error between observed and latent advantages under a reward-noise model. Experiments across multiple mathematical reasoning benchmarks and model scales show that S-GRPO consistently improves performance and stabilizes training. Under controlled reward-noise interventions, it produces smoother optimization dynamics and remains effective even at high noise levels where standard GRPO shows little meaningful learning progress. Our code is available at https://anonymous.4open.science/r/S-GRPO-5F46.