Ivan Kruzhilov

Disentangling Optimization Scale from Preference Scale in DPO

This work shows that $\beta$ entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size, and proposes a centered-softplus reformulation that is argmin-equivalent to DPO for $\beta>0$, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable.