Jul 2026· 2026 23rd International Conference on Ubiquitous Robots (UR)· pp. 347-354· 0 citations· 25 references
Computer Science
Abstract
Large Language Models (LLMs) have demonstrated strong generalization and reasoning capabilities across a wide range of domains, including embodied decision making and robotics. Despite this progress, existing reinforcement-based embodied agents often struggle with novel tasks requiring complex spatial reasoning and multi-object manipulation when relying solely on the robot’s egocentric view. In this paper, we propose SPaRL (Spatially-aware Reinforcement Learning with Language), a novel framework that explicitly integrates a structured 3D scene graph into an LLM-based reinforcement learning policy. By representing objects and their spatial relationships in a compact, semantically meaningful graph and conditioning it on task instructions, SPaRL provides the policy with explicit relational context beyond raw visual inputs. The scene graph is dynamically pruned to retain instruction-relevant objects and relations, serialized into natural language, and jointly processed with visual observations and task descriptions by a frozen LLM backbone. We evaluate SPaRL on language-conditioned rearrangement tasks in Habitat. Results show that incorporating an instruction-conditioned scene graph consistently improves performance over a vision-only LLM policy. Across curriculum training, SPaRL achieved improved performance on the Language Rearrangement benchmark. These results suggest that explicit spatial representations provide useful inductive bias for object reasoning, particularly as task difficulty increases.
Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments. Modern vision-language models (VLMs) already encode spatial priors for visual grounding, spatial reasoning, and pointing, but these capabilities are rarely elicited directly for robot control. Existing navigation systems instead rely on task- or embodiment-specific components, fragmenting perception, reasoning, and action while offering limited generalization. Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads. LightNav-0 represents diverse navigation tasks through a unified token interface: dual-channel pointing expresses task-, scene-, and embodiment-agnostic spatial intent, while a residual vector-quantized action tokenizer maps this intent to precise, embodiment-specific trajectories. Together with temporally aware visual history compression, ER mid-training, supervised fine-tuning, and reinforcement learning, this formulation supports instruction following, open-vocabulary object navigation, and visual tracking within a single model. The navigation training corpus spans 2K+ scenes and 4K+ hours of embodied navigation data. LightNav-ER, the embodied-reasoning checkpoint used to initialize LightNav-0, attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves state-of-the-art monocular success rates across all 10 public navigation simulation settings. Real-world evaluations further demonstrate zero-shot generalization across robot embodiments, diverse scenes, and static and dynamic targets. These results establish compact VLMs as a unified and transferable backbone for generalist embodied navigation.
Shao-An Wang, Ao-Cheng Luo, Fei Huang et al.· 0 citations
Embodied navigation is a core task in embodied AI. It requires comprehensive scene understanding and precise spatial reasoning. Recent vision-language models (VLMs) with strong generalization capabilities and rich commonsense knowledge have shown remarkable performance when applied to embodied navigation tasks. However, these models still encounter insufficient understanding of 3D geometry and spatial semantics when applied to real-world 3D navigation. To address this, we propose CoNav, a collaborative cross-modal reasoning framework. First, we pretrain a 3D-language model with the curriculum learning schedule and prepare a pretrained vision-language navigation agent. Next, with lightweight fine-tuning on a small 2D-3D-text corpus, the vision-language navigation agent learns to combine visual evidence with knowledge from the 3D-language model. Finally, the pretrained 3D-language model communicates with the vision-language navigation agent, enabling collaborative cross-modal reasoning and resolving ambiguities during navigation. This yields more reliable and efficient image-3D fusion for embodied navigation. CoNav introduces a new collaborative framework between a 3D-language model and a vision-language navigation agent for embodied navigation. Notably, CoNav requires only a small 2D-3D-text corpus to align 3D and 2D data. CoNav achieves clear improvements on four standard embodied navigation benchmarks (R2R, CVDN, REVERIE, SOON) and two spatial reasoning benchmarks (ScanQA, SQA3D). Moreover, Under similar success rates, it also finds shorter paths than prior methods, as measured by SPL. The results demonstrate the value of collaborative 2D-3D reasoning for embodied navigation.
Haihong Hao, Mingfei Han, Changlin Li et al.· IEEE Transactions on Pattern...· 0 citations
For robots to operate reliably in real-world environments, they need to perceive their surroundings, act, and reason about the consequences of those actions. Rapid progress in the domains of representation learning, VLA models, and world models has significantly enhanced the capabilities of robot learning systems, enabling robots to work in increasingly complex environments. However, these paradigms are typically developed in isolation, resulting in fragmented systems that struggle with generalization, long-horizon temporal reasoning and planning, and deployment in unstructured environments. In this survey, we present a unified perspective on robot learning by organizing the existing methods along three complementary axes: understanding through representation learning, acting through VLA models, and reasoning through world models. We introduce a structured taxonomy that captures key design choices in environment representation, policy learning, and predictive modeling, and summarize the recent progress in these domains. Beyond classifying the existing works, we analyze how these components interact, discuss common limitations, and highlight emerging trends towards more integrated systems. Through this lens, we identify the challenges in the domain of robot learning, including uncertainty quantification, out-of-distribution generalization, cross-embodiment transfer, long-context understanding, and long-horizon planning. We argue that these challenges arise not only from limitations within individual components but also from the lack of integration across perception, action, and reasoning. Building on this analysis, we outline future directions towards unified, physically grounded, and probabilistic robot learning to develop robust robotic systems that maintain consistent internal representations and support decision making over extended interactions in real-world environments.
Shaunak A. Mehta, Ananya Hazarika, Hao-Chen Zhang et al.· Trans. Mach. Learn. Res.· 0 citations
WorldScape Policy 2.0 is introduced, a controllable WAM with reasoning-augmented long short-term memory and fine-grained instruction following and in-context adaptation that demonstrates superior capabilities in long-horizon autonomous planning, fine-grained instruction following and in-context adaptation.
Hai-Sheng Su, Zong-Dai Liu, Xin Jin et al.· arXiv.org· 2 citations
Effortless object finding by humans, even in cluttered or unseen environments, relies on the seamless integration of perception, memory, and contextual inference. In contrast, embodied robots operating under egocentric perception and partial observability frequently struggle with dynamic spatial relations and long-term consistency, leading to inefficient, repetitive search behaviors. Here we present Human-like Memory Navigation (HM-Nav), a brain-inspired architecture that bridges this cognitive gap by integrating transient sensory inputs with an evolving internal model to enable long-term object navigation. HM-Nav employs three synergistic pillars: (i) Perception: a multi-view fusion module that reconciles viewpoint inconsistencies into unified representations; (ii) Memory: an adaptive dynamic knowledge graph that accumulates semantic-spatial associations over extended timescales; and (iii) Inference: an experience-driven trajectory optimization mechanism that learns from past failures to suppress cyclic and suboptimal search patterns. Validated in simulations and real-world trials, HM-Nav demonstrates superior navigation performance and robust sim-to-real transfer, significantly outperforming existing benchmarks. Our findings suggest that emulating human-like memory structures is essential for achieving resilient, long-term autonomy in complex, open-ended environments.
Vision-Language-Action (VLA) models encode visual observations as flat 2D patch tokens that carry no intrinsic geometric structure, and augmenting them with dense monocular depth injects per-pixel scalar values that encode neither surface orientation nor geometric confidence. This leaves the policy with limited structured spatial reasoning for action prediction. We propose GaussVLA, a Mamba-based VLA that incorporates two custom modules: Gaussian Spatial Tokenizer (GST) to lift frozen semantic and depth features into compact 3D Gaussian tokens, pools geometrically salient regions with learned queries, and \emph{Depth-Aware Chain-of-Thought (DA-CoT)} that performs structured, non-autoregressive geometric reasoning under language and flow-time conditioning. Across both simulation and real-world evaluations, GaussVLA demonstrates strong spatial-manipulation performance while remaining parameter-efficient. On LIBERO, it achieves 93.5% average success and 100.0% success on the Spatial suite with only 200M parameters, improving over SpatialVLA by 19.7% relative average success while remaining significantly more parameter-efficient.
MD SELIM SAROWAR, Md Tanvir Islam, Sungho Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
