Skip to content

ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild

Sep 2026 · arXiv.org · Vol abs/2609.34817 · 0 citations · 49 references
Computer Science

TL;DR

ESTHER is proposed, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo, and shows state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods.

Abstract

Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.

View source

Similar papers

Conference Aug 2026

Partial Human Perception from a Robot's Egocentric Visual Perspective: A Benchmark for Body-Part Visibility Prediction

Invariant human perception from visual input is essential during human-robot interaction. Although multiple approaches exist, such as face recognition or human pose estimation, these methods frequently fail in real-world interactive settings, where robots typically operate from close-range view-points that result in pa...

Anna L. Lange, Bo-Song Ding, M. Kirtay et al. · 0 citations
Preprint Sep 2026

Seeing the World and the Self from Egocentric Video

Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation.

Kai Guan, Minchao Jiang, Ruichen WangLi et al. · 0 citations
Sep 2026

IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining

Egocentric human video offers a scalable data source for dexterous manipulation, yet using it to train humanoid robots presents two challenges: (1) an embodiment gap, as human hands differ structurally from robot end-effectors and low-cost egocentric recordings lack the torso kinematics required by conventional retarge...

Hui-Min Pan, Y. Ren, Kun-Peng Song et al. · 0 citations
Sep 2026

EgoWild2Dex: Learning Dexterous Robotic Manipulation from In-the-Wild Human Experience

EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, is introduced and GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot...

Kun-Li Channing Lin, Xu-Tao Wen, Jing Lin et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.