Jul 2026· IEEE Transactions on Image Processing· Vol 35, pp. 8246-8259· 1 citation· 65 references
Computer ScienceMedicine
Abstract
To facilitate smart wearable devices or human-like robotics with real-time first-person perspective perception ability, recent researchers proposed the Egocentric Online Action Segmentation (EOAS) task. It requires models to recognize what is happening in egocentric streaming videos and discriminate the starting and ending times of an activity in a real-time manner. However, compared with offline-recorded exocentric videos, egocentric streaming videos cannot provide equivalent sufficient temporal-spatial cues due to the limited perspective and unknown coming frames. Hence, it raises a high demand for the long-term episodic memory ability of models. To this end, most previous approaches work on compressing long-term memory into feature representations. In this paper, we propose a novel EOAS paradigm, termed Parametric Context Memory Learning (PCML), which integrates episodic memory into learnable parameters and keeps dynamic updates according to real-time frames. Concretely, we design the Parametric Context Perception layer and construct a novel Episodic Semantic Memorization Network (ESMN) based on it, which integrates episodic memory into learnable parameters and keeps dynamic updates with real-time frames. We evaluate our proposed method on three public egocentric streaming video benchmarks including EgoPER, EgoProceL, and GTEA. Extensive experiments demonstrate the ESMN model significantly outperforms recent state-of-the-art methods. Our code is available at https://github.com/XunCHN/PCML
Method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data, is presented, demonstrating that the synthesized data substantially improve downstream WAM generalization.
Zexuan Yan, Yuzhou Wu, Yue Ma et al.· arXiv.org· 0 citations
Experiments on egocentric video benchmarks show LogFA significantly improves model generalization to unseen environments while maintaining low computational and data collection costs.
Evaluation of state-of-the-art open-source and closed-source MLLMs reveals that current MLLMs function as lossy summarizers rather than faithful memorizers, highlighting the need for architectures with genuine long-term spatiotemporal memory.
Weitao Chen, Jiaxing Hu, Xie Tianyidan et al.· 0 citations
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
T. Ding, Zhen Luo, Yixuan Yang et al.· 0 citations
Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.
Anubhav Gupta, A. Kambhamettu, Vatsal Agarwal et al.· 0 citations
Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
Zhe Liu, Jinghua Hou, Yuxiang Lu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
