This work proposes three exact old-logit acquisition strategies: snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption, and compares their system trade-offs to adopt a revised PPO-EWMA method.
Zhong Guan, Yong-Jian Guo, Haoran Sun et al.· arXiv.org· 4 citations· ⚡1
RL-VLA is the first fully asynchronous RL training framework tailored specifically for the system-level challenges of VLA training, and achieves throughput improvements of up to 85.2\% over synchronous baselines while maintaining identical sample efficiency, with scalability validated from 8 to 256 GPUs.
Haoran Sun, Yong-Jian Guo, Zhong Guan et al.· 7 citations
This work proposes Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates, establishing a new standard for robust LLM realignment.