ST-Policy: Encoding Spatiotemporal Continuity for Diffusion Policy

Abstract

Diffusion-based visuomotor policies are a powerful paradigm for robotic imitation learning, yet their sequence models apply a uniform inductive bias at every position of the action sequence, ignoring the spatiotemporal continuity inherent in robotic motion. We identify a fundamental limitation of causal state space models (SSMs): their exponentially decaying effective receptive field induces a positional bias that allocates modeling capacity non-uniformly across action dimensions. We therefore propose ST-Policy, a Mamba-based 3D diffusion policy that encodes this continuity through two complementary mechanisms: (1) zero-parameter scan diversification, which applies deterministic, morphology-aware permutations across SSM blocks to equalize the worst-case effective distance between all position pairs; and (2) lightweight gated smoothing, which injects trajectory continuity priors through Gaussian-initialized depthwise convolutions at roughly 300 parameters. Both are hosted in a dual-branch Mamba mixer that decouples local feature preservation from long-range recurrence. Experiments on 37 simulation tasks and a real-world robot platform demonstrate consistent improvements over prior 3D diffusion policies.