SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
This paper introduces a novel reward-model-free, critic-free, and gradient-based PbRL algorithm compatible with segment preferences named Segment Pairwise Proximal Policy Optimization (SP3O), and provides a theoretical basis for the algorithm and analyze the tradeoff in choosing the segment length.