J. Stastny

Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking

It is shown that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM.