Skip to content
Book Open access

S2WAM: A Self-Supervised World Action Model from Sequential Multimodal Observations for Autonomous Driving

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 41 references

Abstract

Recent autonomous driving systems have increasingly shifted from modular pipelines to end-to-end learning paradigms. However, purely end-to-end approaches are prone to shortcut learning or representation collapse, often requiring auxiliary perception or prediction tasks to improve generalization. These auxiliary objectives typically rely on expensive perception annotations, limiting scalability to large-scale unlabeled data. Moreover, many existing designs are largely heuristic, lacking principled theoretical guidance. In this paper, we present S2WAM, a Self-Supervised World Action Model for autonomous driving that learns directly from sequential sensor observations without requiring perception labels. S2WAM learns a latent world representation by reconstructing global multimodal observations from local partial observations. Additionally, we provide a theoretical formulation of S2WAM as a variant of variational autoencoders, offering principled guidance for model design. As a self-supervised framework, S2WAM can be trained on large-scale unlabeled driving data and produces representations that are effective for downstream planning. Experimental results on the NAVSIM benchmark show that our S2WAM achieves competitive performance without perception supervision, while also demonstrating its capability in 4D scene reconstruction and generation. Code is released at https://github.com/Say2L/S2WAM.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.