Modern large language models

Abstract

Observation: Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the context We identify and characterize a failure mode in large language models aligned with RLHF. We show that inserting a long, irrelevant text prefix that does not contain instructions causes a persistent shift in the model’s activations. This shift, which persists throughout the session, decouples subsequent behavior from the safety constraints established during training. The model begins to exhibit behavioral characteristics consistent with its pre-trained distribution: the failure rate decreases, stylistic constraints disappear, and the tone of responses changes. It is important to note that this occurs without explicit adversarial instructions and without the model agreeing with the prefix’s content. We call this effect “context-induced activation shift.”