Diagnosing Faults in Reinforcement Learning Simulators and World Models with Canonical Polynomial Invariants

Abstract

A large literature builds physical structure into learned dynamics on the premise that models respecting the underlying physics predict better. This work tests that premise using exact polynomial invariants recovered from trajectories and canonicalised as reduced Gröbner bases over ℚ. On Acrobot, exactness provides little benefit for prediction: a consistency regulariser reduces algebraic residual while leaving rollout fidelity essentially unchanged, and one-step validation error remains the better predictor of long-horizon accuracy. Likewise, a shaping potential recovered from a system with a 100% mass error accelerates learning as effectively as the correct potential, indicating that shaping depends on potential geometry rather than physical correctness. Exact canonical invariants instead prove valuable for diagnosis. This paper develops two procedures: screening, which identifies the violated physical constraint, and attribution, which recovers the faulty invariant and identifies the responsible physical parameter. Two algorithmic components enable this diagnostic: normal-form deflation, which removes algebraically trivial multiples exactly, and quotient-space recovery, which avoids tolerance-based nullspace dimension estimation. Across fifteen injected faults, screening localises every broken constraint with no observed false alarms, whereas observation-space baselines localise none; attribution recovers the responsible parameter on all seven single-parameter faults. Paired difference tests detect all faults, showing that the advantage is localisation rather than detection. Perturbing reference generators by 10⁻⁴ preserves 14–15/15 localisations, showing that screening does not require exactness, whereas ideal-equality decisions distinguish perturbations of only 10⁻¹², showing that exact canonical representations are required for algebraic comparison. Applied to 350 release pairs across eleven RL environments, the diagnostic finds no evidence of changed simulator dynamics, instead revealing properties of the benchmark implementations themselves. Code and reproducibility: https://github.com/TesfayZ/algebraicRLtest.git