Skip to content
Open access

Mitigating Think-Answer Mismatch in large language model reasoning through noise-aware advantage reweighting

Oct 2026 · Memetic Computing · Vol 18 · 0 citations · 49 references

Abstract

Group-Relative Policy Optimization (GRPO) is a key technique for training large reasoning models, but its reliance on outcome rewards makes it vulnerable to Think-Answer Mismatch, where final-answer correctness does not faithfully reflect reasoning quality. This vulnerability is amplified in highly imbalanced response groups, where a single corrupted reward can strongly distort group-relative advantages. To address this problem, we propose Stable Group-Relative Policy Optimization (S-GRPO), which derives a closed-form, noise-aware weight by minimizing the expected squared error between observed and latent advantages under a reward-noise model. Experiments across multiple mathematical reasoning benchmarks and model scales show that S-GRPO consistently improves performance and stabilizes training. Under controlled reward-noise interventions, it produces smoother optimization dynamics and remains effective even at high noise levels where standard GRPO shows little meaningful learning progress. Our code is available at https://anonymous.4open.science/r/S-GRPO-5F46.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.