This work presents JITOMA (Just-In-Time On-demand Memory Activation), a closed-loop framework that unifies task reasoning, perception, and memory into a just-in-time growth process, and introduces JITOMA-Bench, a comprehensive suite for long-horizon multi-tasking and complex multi-step reasoning.
Abstract
While 3D Scene Graphs (3DSGs) provide crucial structured representations for embodied agents, conventional Ahead-of-Time, build-everything-then-filter pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand Memory Activation), a closed-loop framework that unifies task reasoning, perception, and memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering resource-intensive operations -- such as dense node captioning and functional inference -- exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade-offs, we introduce JITOMA-Bench, a comprehensive suite for long-horizon multi-tasking and complex multi-step reasoning. Extensive experiments demonstrate that JITOMA substantially reduces active graph size and captioning latency, while maintaining stable processing time under long-horizon task switching.
Robots are now expected to execute increasingly complex long-horizon tasks in unstructured environments. Despite the strong potential of pretrained Vision-Language Models (VLMs) in task planning, their direct application to robotic manipulation is hindered by logical reasoning deviations and inadequate geometric scene perception. This work proposes an adaptive task planning method based on video priors and dynamic scene graphs (ATP-VPDSG). It leverages the VLM to extract manipulation logic from video demonstrations, thus supplementing manipulation priors. Meanwhile, scene graphs were integrated to convert unstructured environments into structured representations with spatial topological relations, compensating for perceptual deficiencies. A dual-track feedback mechanism based on visual expectations was further incorporated to enable failure diagnosis and adaptive replanning in complex environments. Extensive long-horizon robotic manipulation experiments were conducted on the LIBERO-10 benchmark with Qwen3-VL as the core VLM. Results showed that ATP-VPDSG achieved an average task planning accuracy of 91.2% and a task execution success rate of 74.67%, outperforming the selected task planning baselines. Ablation studies verified that video priors and dynamic scene graphs exerted complementary effects on logical constraints and physical feasibility. Furthermore, a real-robot experiment on an industrial slider–rail assembly task demonstrated successful sim-to-real transfer, achieving an 82.0% success rate without task-specific fine-tuning.
Guang-Hui Ma, Jia-Hui Guo, Xin-Hua Tang et al.· Italian National Conference...· 0 citations
A modular mapping architecture is demonstrated that establishes 3D Semantic Scene Graphs (3DSSGs) as its foundational back-end, enabling the dense representation of extensive environments containing thousands of unique object instances and supporting open-vocabulary queries via CLIP features without requiring any additional post-processing steps.
F. Igelbrink, Lennart Niecksch, Martin G. ̈unther et al.· 0 citations
Prior-SG achieves state-of-the-art semantic region segmentation accuracy compared to recent baselines, robustly delineates distant functional boundaries in the absence of physical walls, and uniquely provides zero-shot ontological flexibility, enabling the robot to entirely restructure its spatial partitioning based on a given high-level task.
G. Tonetti, Laurent Kneip, Abel Gawel et al.· 0 citations
Continuous-environment vision-and-language navigation (VLN-CE) requires interpreting natural-language instructions in unseen 3D environments and executing continuous low-level actions. Existing methods often depend on LiDAR, panoramic cameras, or extra sensors; separate geometric-mapping and semantic-navigation visual representations can cause long-trajectory spatial-semantic inconsistencies. We propose LG-VLN, a monocular zero-shot framework with shared visual features and LangGraph-based state orchestration. An online feed-forward 3D reconstruction network predicts depth, camera poses, and dense point clouds for agent-pose estimation and global map fusion. Geometry and navigation share dense CleanDIFT features: semantic consistency rejects incorrect inter-frame correspondences, while target-instance constraints define visual references whose similarity combines with local BLIP-2 image-text relevance to form a semantic value map. LangGraph represents instruction parsing, geometric perception, semantic value updates, path planning, action execution, and failure recovery as a directed state graph with conditional transitions, persistent state, and modular recovery mechanisms. On a fixed 550-episode subset of the R2R-CE val-unseen split, LG-VLN achieves 21.3% success and 12.1% success weighted by path length. Ablations show shared semantic features improve navigation, further boosted by combining visual similarity and image-text relevance. Results establish shared visual representations and explicit state orchestration as effective for zero-shot VLN-CE using monocular RGB alone. Code will be publicly released for reproducibility.
Jian-He Zhao, Yan-Hua Qiu, Zhi-Yu Zhang et al.· 0 citations
HAM-VLN is presented, a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph and reduces the context length by more than 65% compared to previous methods.
An Liu, Bingxi Liu, Hongyu Ding et al.· arXiv.org· 0 citations
Planning in complex environments requires task specifications grounded in representations that capture objects, relations, and affordances; scene graphs meet this need, but their size in large environments hinders efficient planning. While task-aware pruning and hierarchical abstractions have been explored, a general, task-centric formalization of what constitutes a sufficient scene graph for planning remains open. This paper provides such a formalization by modeling planning over scene graphs within an information-spaces framework through the definition of scene graph transition systems and relevant action semantics for navigation and manipulation. We then introduce derived scene graphs via information mappings that merge and prune nodes and induce quotient transition systems augmented with motion primitives to capture higher-level actions over merged graph nodes. Sufficiency is characterized by two conditions: (i) the information mapping yields a deterministic quotient, and (ii) the task is well-posed over derived traces, ensuring plans found on the derived model are feasible on the maximal system. We illustrate the framework using a task over an example environment, showing both sufficient and insufficient reduced scene graphs.
Basak Sakçak, Francesco Verdoja· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
