Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
The Entropy-Scaled Trust Region (ESTR), which scales each token’s off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection, consistently outperforms existing asynchronous methods and achieves the best train-inference consistency.