Policy Drift in Learning AI Agents: A Dynamical Systems Perspective on Security Degradation

Abstract

Paper 1 of this series introduced clean attacks adversarial inputs that satisfy all observable policy constraints yet redirect autonomous agent behavior away from operator intent. Still, uncertainty remains about how such shifts unfold across extended usage. Even though immediate effects suggest weakened breaches cause less impact (58.1%) compared to full goal swaps (64.8%), real-world cases often describe worsening divergence after multiple cycles, hinting at gradual breakdowns needing deeper modeling. Here, deviation over time emerges as slow but compounding movement of an agent’s actual choices beyond intended boundaries Π₀ due to recurring subtle assaults. This shift follows complex patterns - not linear decay, rather like systems building hidden strain until sudden collapse occurs past threshold t, then veering sharply into attacker-influenced regimes Π without return. Beyond tipping point λ in frequency of these disguised interventions, recovery fades; alignment dissolves not by force, but through accumulation. This study offers four core advances. Right away, policy drift takes shape through a nonlinear differential equation - framed within the policy state space - with support from Lyapunov stability concepts alongside bifurcation methods. Instead of vague metrics, the Intent Drift Rate appears: a concrete number per dialogue turn, built as the time-based change in the Behavioral Drift Index first laid out in Paper 1. Following that, Π₀’s resilience gets examined via a two-valley energy model, uncovering how the tipping point labeled λ controls whether systems rebound or shift into Π. Later still, validation arrives through tests on AegisBench-MT - a version expanded to 480 conversations - spanning three types of agents plus four leading large language models, spotting clear shifts near t ≈ 9 steps along with a split triggered around λ ≈ 0.42. Foundational input comes from IDR, shaping core logic across Papers 3 through 5. Trust decisions driven by user purpose rely on it, just as node-level shift tracking within agent networks does. Response mechanisms that activate when deviations occur also build upon this base. Each test applies common reference terms defined earlier in Paper 1 - elements like φ, Π, SVS, and BDI remain consistent throughout.