Preference Optimization for Open-Weight Language Models: A Survey and Low-Resource RLHF Case Study

Abstract

Open-weight language-model post-training combines preference learning, reinforcement learning, verifiable feedback, and increasingly, agentic training and on-policy distillation. Comparing these systems requires separating their optimization algorithms from their feedback sources, sampling procedures, and evaluation targets. We present a comparative survey of representative methods and releases, with a targeted literature update through 18 September 2026. The review distinguishes foundational PPO, DPO, and GRPO methods from later developments in mixed feedback training, multimodal agents, asynchronous reinforcement learning, and cross-stage capability consolidation. We complement this synthesis with SAWYER, a single-T4 GPT-2 case study implementing supervised fine-tuning, Bradley–Terry reward modeling, and 300 PPO updates. The reward model achieves 55.3% held-out preference accuracy. Across 100 evaluation prompts reported as unseen during training, mean learned reward increases from 1.354 for the supervised policy to 1.499 after PPO, but the proxy comparison yields 41 PPO wins, 20 ties, and 39 losses. In a selected 25-example audit, a separate AI judge prefers the supervised policy in 15 comparisons, including nine for which PPO receives higher learned reward. These observations reveal a mismatch between aggregate proxy improvement and selected-example judgments, without establishing a population-wide quality decline or a causal exploitation mechanism. A strongly negative logged KL-related statistic further motivates implementation-level diagnosis. Together, the survey and case study connect contemporary post-training design choices to a practical evaluation principle: optimization success, feedback reliability, and independently assessed behavior require distinct evidence.