On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models
It is revealed that off-the-shelf LRMs exhibit a confidence prior heavily concentrated on a few high values, which persists throughout RL, and it is proved that this concentration suppresses policy gradient updates for rarely sampled confidence values and inflates the lower bound on expected Brier risk.