The Entropy-Scaled Trust Region (ESTR), which scales each token’s off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection, consistently outperforms existing asynchronous methods and achieves the best train-inference consistency.
BRACE is proposed, an anchored Bellman-residual correction for stale value models that bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation.
Guan-Qun Zhao, Zi-Jun Xie, Bin-Bin Zheng et al.· 0 citations
Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts, and refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction.