Skip to content

Know Your Body: A Harness for Direct and Self-Improving Robot Control with VLMs

Sep 2026 · arXiv.org · Vol abs/2609.28530 · 1 citation · 52 references
Computer Science

TL;DR

KnowBody is introduced, a harness that makes action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen, and knowledge dependent on revised body estimates is rechecks before reuse.

Abstract

A general-purpose vision-language model can understand a task goal without knowing how a particular robot's motion and functional parts produce the intended effect. We introduce KnowBody, a harness that makes these action-relevant body relations explicit, queryable, and revisable while keeping the model weights frozen. Initialized from one off-task trajectory, a partial body model guides action selection and the interpretation of past interactions. New evidence refines the model, and knowledge dependent on revised body estimates is rechecked before reuse. Across 32 fixed-budget trials on four real-robot tasks, initialized KnowBody achieves 75% completion versus 25% for the native harness and requires fewer planner rounds on successful trials in tasks completed by both. With persistent updates enabled, planner rounds decrease by 29-53% from the first to the fifth recorded success.

View source

Similar papers

Preprint Oct 2026

CarryOn: A Dual-Loop Harness for Personalized and Reliable Robot Assistance

CarryOn is a dual-loop harness designed to augment a frozen VLA policy, enabling personalized task selection and execution grounding across conversations and changing scenes, and increases recovery success from 13% to 80% when refined skills are reused.

Zixuan Wang, Yu-Qi Liu, Yu-Xin Chen et al. · 3 citations
Preprint Nov 2026

HITCH: Teaching a Vision-Language Model to Operate a Humanoid through a Whole-Body Harness

Enabling vision-language models (VLMs) to control humanoid robots requires both reliable whole-body interfaces and knowledge of how to use the body that carries over across tasks. We present HITCH (Harness Interfaces To Command Humanoids), a harness in which perception returns the 3D position of any point the VLM selec...

Qing-Zhou Lu, Chen-Xi Liu, Xin-Ming Wang et al. · 0 citations
Preprint Oct 2026

RoboDoctor: Training-Free Recursive Self-Improvement for Robot Policies

Generalist robot policies and language-model agents provide complementary capabilities, but completing a task also requires adapting their use to the consequences of physical interaction. We study execution as a locus of adaptation: feedback guides the current action and supplies evidence for revising subsequent decisi...

He-Yang Yu, Hao-Min Li, Si-Hang Li et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

Building reliable robot capabilities across diverse tasks requires substantial human effort to develop and maintain skills, design rewards, and integrate perception with control. We present Reconstruct, Practice, Go Real (RPG), a framework for autonomous improvement of robot execution systems without updating model wei...

Yen-Jen Wang, Hao-Zhe Jiang, Shu-Ying Deng et al. · 11 citations
Sep 2026

RoboHarn-Evo: Evolving Hierarchical Physical Knowledge for Self-Improving Robotic Manipulation

RoboHarn-Evo is introduced, a dual-loop harness that evolves Hierarchical Physical Knowledge (HPK) from physical experience that demonstrates that physical interaction can be accumulated into reusable knowledge for improving subsequent manipulation.

Shi-Feng Bao, Fan-Ding Huang, Yi-Han Lin et al. · 4 citations
Preprint Oct 2026

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution...

You-Wei Zhou, Bo-Han Hou, Yan-Kai Fu et al. · 3 citations · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.