Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that...
Hong-Yang Li, Xiao Li, Caesar Wu et al.
· 0 citations