Skip to content

SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.22085 · 2 citations · 41 references
Computer Science

TL;DR

This work introduces SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask, which substantially improves best-of-N policy steering on real-world manipulation tasks.

Abstract

Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-N policy steering.

View source

Similar papers

Sep 2026

StageGuard: Learning Stage Transitions for Long-Horizon Robot Tasks via Agentic Distillation

Hierarchical planning frameworks combine skills from multiple robot control policies for long-horizon task execution, where determining when to terminate the current skill and advance to the next subtask is essential. Existing approaches often rely on pre-designed completion signal checkers that are hard to obtain in r...

Jinbang Huang, Yuan Hu, Zhi-Yuan Li et al. · 1 citation
#machine learning Preprint Sep 2026

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at bottlenecks while allowing training rollouts to proceed with minimal human intervention, is presented.

Sichang Su, B. Yang, Zhi-Yun Deng et al. · 0 citations
#artificial intelligence Preprint Sep 2026

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier by executing each task's own verifier.

Jun-Yao Yang, Yu-Cheng Shi, Zhong-Zhi Li et al. · 9 citations
Sep 2026

FIERCE: From Generalist Robot Policies to Fast Specialists via Progress-Failure Feedback

FIERCE is presented, a generalist-initialized reinforcement learning framework centered on a unified, task-adaptive progress-failure evaluator that separates feedback quality, policy-learning efficiency, and deployment cost across simulation and two contact-rich real tasks.

Run-Jia Tan, Yuang Tu, Yu-Jie Yan et al. · 0 citations
#machine learning Preprint Oct 2026

TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery

Emerging long-horizon agentic tasks require repeated model calls, worsening the inference cost of already-costly language models. While narrow agentic tasks suggest potential for aggressive model pruning without performance drop, empirical results show existing methods proposed for question answering tasks severely deg...

Yuan-Zhe Li, Peng-Xin Wang, Yuxin Ren et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

Recent advances in robot learning have enabled generalist control policies capable of completing a wide range of tasks. However, their performance degrades when deployed in unseen environments, making it critical to detect failures and teach recovery behaviors. Existing runtime monitoring methods often require task- an...

Ryan Li, Yigit Korkmaz, Erdem Biyik · 2 citations · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.