RL Starts before RL: On Policy Distillation for Better Reinforcement Learning
The results support OPD as preparation for RL and favor forward KL when OPD is followed by RL in the authors' comparison, which highlights the importance of both objective choice and the states receiving supervision for subsequent RL.