Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization
The findings suggest that the teacher models used to generate preference data can interact with alignment training objectives in unexpected ways, generalizing to undesirable and potentially harmful behaviors like sycophantic agreement.
C. Blank, Z. Ying, Christopher Potts et al.
· 0 citations