S2WAM: A Self-Supervised World Action Model from Sequential Multimodal Observations for Autonomous Driving
Abstract
Recent autonomous driving systems have increasingly shifted from modular pipelines to end-to-end learning paradigms. However, purely end-to-end approaches are prone to shortcut learning or representation collapse, often requiring auxiliary perception or prediction tasks to improve generalization. These auxiliary objectives typically rely on expensive perception annotations, limiting scalability to large-scale unlabeled data. Moreover, many existing designs are largely heuristic, lacking principled theoretical guidance. In this paper, we present S2WAM, a Self-Supervised World Action Model for autonomous driving that learns directly from sequential sensor observations without requiring perception labels. S2WAM learns a latent world representation by reconstructing global multimodal observations from local partial observations. Additionally, we provide a theoretical formulation of S2WAM as a variant of variational autoencoders, offering principled guidance for model design. As a self-supervised framework, S2WAM can be trained on large-scale unlabeled driving data and produces representations that are effective for downstream planning. Experimental results on the NAVSIM benchmark show that our S2WAM achieves competitive performance without perception supervision, while also demonstrating its capability in 4D scene reconstruction and generation. Code is released at https://github.com/Say2L/S2WAM.