How to post-train on a surrogate: Envelope sampling mitigates reward hacking
Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable sid...
Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan et al.
· 0 citations