Skip to content

What Can a Recurrent State Safely Forget?

Sep 2026 · 0 citations · 14 references
Computer Science

TL;DR

An auditable finite-future framework that extends guarantees to continuous futures via an explicit completeness modulus is developed, and Controlled experiments validate the certified margins, scaling laws, and automated probe refinement under a safety-first evaluation paradigm.

Abstract

Recurrent models must preserve information that changes future behavior while suppressing hidden-state error. These objectives conflict: contraction improves stability, but contraction along a future-distinguishing direction destroys memory. We formalize this boundary through the predictive quotient of a recurrent state space. Two hidden states are equivalent when they induce the same conditional future; their equivalence classes form predictive fibers. Every exact semantics-preserving corrector acts as the identity on this quotient. At a regular point with hidden dimension d and predictive dimension k, it can eliminate at most d - k independent directions. This establishes a discrete-continuous boundary: finite predictive states admit positive-radius exact correction basins, whereas an uncountable continuum of future-distinguishable states cannot be decoded after arbitrary positive-radius perturbations in finite-dimensional Euclidean space. To operationalize this principle, we develop an auditable finite-future framework. A compact deployment bank W is evaluated against an independent audit bank A (W subseteq A) on a declared correction domain. Under generative probe access and audit-metric coverage, finite stochastic rollouts furnish a high-probability certificate for the separation margin Omega_{W|A}(delta). Preserving learned W-predictions within this certified margin guarantees bounded audit-semantic distortion. For intrinsic audit dimension k, the required probe outcomes scale as O(M * Omega^{-(k+2)}), where M = |A|; a matching minimax lower bound proves this exponent is optimal. Extending guarantees to continuous futures is achieved via an explicit completeness modulus. Controlled experiments validate the certified margins, scaling laws, and automated probe refinement under a safety-first evaluation paradigm.

View source

Similar papers

#machine learning Preprint Oct 2026

Prediction Is Not a Sufficient Memory State

A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension e...

Robert Jenkinson Alvarez · 0 citations
Preprint Sep 2026

Hidden-State Updates and observable-record composition in retrocausal models

A hidden-variable model can reproduce context-independent individual readouts while failing to preserve observable records when devices are connected or reused. We distinguish two composition requirements. For a finite family of nonnegative instrument matrices, an invariant space generated by normalization defects char...

D. Weerasinghe · 0 citations
#machine learning Preprint Sep 2026

Minimal Recurrent Behavioral Memory for Imitation under Partial Observability

This minimal recurrent behavioral memory is characterized by a compatibility relation: under transitivity its classes attain the exact minimum, while the general case is an entropy minimization over closed compatible state assignments, with exact certificates on finite instances.

Xian-Yao Li, Fang Xu, Rui Min et al. · 0 citations
Preprint Sep 2026

Compact but Moving: Intervention-Relevant Geometry in Recurrent World Models

Results show that compact intervention structure can persist as a moving, state-dependent local geometry embedded in high-dimensional recurrent dynamics, without implying a fixed or dynamically closed low-dimensional state.

Yu-Ming Chen, Yang Liu · 0 citations
#machine learning Preprint Oct 2026

When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs

Linear-attention and state-space LLMs carry long context in a fixed-size recurrent state, and quantizing that state is the natural way to cut its cost during long decode. Unlike a weight tensor, the state is rewritten at every step, so a quantizer applied at each step sits inside the recurrence and its errors compound....

Yu-Hang Jiang · 0 citations

World Models with Predictable Long-Horizon Marginals

Three properties of a world model are distinguished: the distribution it approaches, the rate of approach, and the conditional dynamics it learns, which derive an absolute convergence bound from finite initialization banks and control departure from the reference through conditional action-space divergence.

Yu-Hao Du, Shu-Nian Chen · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.