ANCHOR: Taming Entropy Dynamics for Stable and Efficient Reasoning of Large Language Models

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard approach for improving complex reasoning in Large Language Models (LLMs). However, entropy collapse remains a critical bottleneck, often leading to performance degradation. While typically attributed to reduced exploration, we identify a more severe failure mode: update deadlock. We show that policy entropy bounds both the policy gradient and probability update norms; consequently, entropy collapse effectively stops reward signal backpropagation, preventing further policy learning regardless of data quality. To analyze this, we develop a micro-dynamics framework based on single-step logit updates. Our results indicate that entropy evolution depends on the interaction between advantage signals and the log-probability residual relative to its expectation, offering a unified perspective on existing interventions. Based on these findings, we propose ANCHOR, an algorithm designed to resolve deadlocks and maintain stability. ANCHOR uses a dual mechanism: it dynamically anchors the entropy decay of positive samples to prevent premature saturation, and uses entropy variations to stabilize updates for negative samples. Extensive experiments on reasoning benchmarks show that ANCHOR significantly outperforms strong baselines. Unlike heuristic methods that induce verbosity to mitigate collapse, ANCHOR promotes concise reasoning paths, improving both stability and efficiency.