Preprint

Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

Haoyu Zhang Zecui Zeng Bin Wang Lusong Li Liang Lin Long Cheng

Aug 2026 · 0 citations · 37 references

Computer Science

TL;DR

This work introduces Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations that provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives.

Abstract

Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation

This work introduces Robo-Dopamine 2.0, a history- and OOD-aware process reward model with a pairwise prediction interface that combines history-conditioned pairwise rewards that use source-aligned reference panels for synthetic OOD queries and observed rollout history for online queries, while preserving the queried endpoints.

Yijie Xu, Hao-Peng Jin, Run Zhou et al. · 0 citations

Jul 2026

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation, and provides effective reward guidance for downstream model predictive control and reinforcement learning.

Yu Fang, Wanxi Dong, Jiaqi Liu et al. · arXiv.org · 1 citation

Jul 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts, and refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction.

Binbin Zheng, Zi-Jun Xie, Guan-Qun Zhao et al. · arXiv.org · 1 citation

#natural language process... Preprint Sep 2026

Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

Vision-language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions. We show that current VLM reward models often violate this property. Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical robot behavior between failure and success. To measure this failure mode, we introduce ROBORMBENCH, a benchmark with 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites. Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, grows under more divergent rewrites, and is not reliably reduced by scale or explicit reasoning. Dedicated reward models trained with trajectory-grounded supervision are substantially more stable. These results show that paraphrase robustness is a core requirement for reliable VLM-based reward modeling in robotics.

Wonje Jeung, Sangyeon Yoon, Hyesoo Hong et al. · 0 citations

Preprint Aug 2026

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

WorldCycle is introduced, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions.

Bohai Gu, Yueyang Yuan, Taiyi Wu et al. · 2 citations

Jul 2026

H2SD: Hybrid Hindsight Self-Distillation

Experiments on challenging reasoning benchmarks show that H$^2$SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.

Qi Cai, Yi-Chuan Ma, Linyang Li et al. · arXiv.org · 2 citations