RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
TL;DR
This work introduces RedFlow, an offline post-training method for flow-matching VLA policies that requires neither external HIL corrections nor online data collection during post-training, and improves average success across four LIBERO suites.
Abstract
Reinforcement learning (RL) can improve Vision-Language-Action (VLA) policies from deployment experience, but reward- and preference-based RL primarily identifies desirable behaviors without specifying how to correct failed actions, underutilizing failure trajectories and limiting sample efficiency. Can such corrections be derived from fixed rollouts? Our key insight is that rollouts with different outcomes may contain action chunks executed in similar states, enabling higher-quality chunks to provide locally supported corrective references. Building on this insight, we introduce \textbf{RedFlow}, an offline post-training method for flow-matching VLA policies. \emph{Execution-Context Matching} groups chunks using a compact representation of estimated task progress and robot proprioception. \emph{Quality-Guided Action Redirection} assigns signed chunk-quality scores and aggregates higher-quality chunks into corrective targets, reinforcing high-quality chunks, suppressing low-quality chunks, and redirecting correctable chunks toward their targets. RedFlow requires neither external HIL corrections nor online data collection during post-training. Across four LIBERO suites, RedFlow improves average success from 56.2\% to 68.2\%, outperforming the strongest evaluated offline baseline, AWR (62.3\%), by 5.9 points. Across three real-robot tasks, it improves average success from 56.7\% to 74.7\%. On LIBERO-Spatial, RedFlow reaches 75.8\% success with 1{,}536 fixed rollouts, while the evaluated online methods require 8.7--16$\times$ as many fresh post-training rollouts to reach the same threshold.
