RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages.
Abstract
Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.
VINE is proposed, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies and achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task.
Rushuai Yang, Zhuo Han, Houlin Li et al.· arXiv.org· 0 citations
Offline reinforcement learning (RL) aims to derive effective policies from fixed datasets without environment interaction. While generative models such as diffusion and flow matching improve policy expressiveness, they often suffer from high computational cost due to multi-step sampling and limited representational richness from uninformative Gaussian priors. To address these challenges, we propose Latent MeanFlow Policy Optimization (LaMPO), a generative policy framework that formulates offline RL as latent generative policy optimization. LaMPO learns a behavior-conditioned latent distribution, which provides an informative prior and reduces the modeling burden of the generative process. Conditioned on this prior, the MeanFlow policy is trained to predict the average action velocity, enabling high-fidelity one-step action generation that avoids the cost of iterative sampling. Guided by the MeanFlow policy, the target policy is then optimized under a MeanFlow-induced behavior constraint, achieving effective policy improvement. We further establish a theoretical performance lower bound of the target policy relative to the MeanFlow policy. Extensive evaluations across 71 tasks in OGBench, D4RL, and real-world humanoid manipulation demonstrate that LaMPO achieves superior performance with high efficiency, yielding a 19% average improvement over existing state-of-the-art methods and an 81% success rate in real-world robotic tasks.
Tenglong Liu, Xin Xu, Yixing Lan et al.· IEEE Transactions on Pattern...· 0 citations
Offline reinforcement learning is limited by the coverage of the offline dataset, which makes it difficult for policies to generalize to unseen goals and behaviors. We introduce CoPL, a framework for counterfactual offline policy learning via large language models. Unlike prior relabeling approaches that modify only instructions or action annotations, CoPL generates full counterfactual trajectories from real trajectory seeds, including states, actions, and their temporal evolution. Specifically, CoPL preserves the initial state of a real trajectory, rewrites the original goal into a semantically valid but behaviorally distinct counterfactual goal, and synthesizes the corresponding goal-conditioned rollout. The generated trajectories are then repaired and filtered before being merged with real offline data for policy learning. This design enables offline RL to learn from behavior patterns that are absent or weakly represented in the original dataset. Experimental results on robotic manipulation tasks show that CoPL substantially improves generalization on unseen tasks under both semantic and linguistic shifts, including a 55.8% relative improvement on the CLEVR-Robot CF-Easy setting, from 17.2% to 26.8% success rate.
Maosen Zeng, Yunan Liu· 2026 IEEE International Conf...· 0 citations
Offline reinforcement learning aims to learn effective policies from fixed datasets without online interaction, necessitating conservative constraints to mitigate the out-ofdistribution issue. Although existing approaches alleviate this issue through conservative constraints or policy regularization, they still struggle to effectively exploit high-quality samples in offline datasets. To address these issues, this study proposes Dual Advantage-Guided Offline Reinforcement Learning (DAG). The approach designs an Advantage-Guided Variational Autoencoder (AG-VAE) to reconstruct the behavior policy from offline datasets. The policy learns from both reconstructed actions and high-advantage dataset actions, with advantage weighting facilitating policy generalization. To evaluate the effectiveness of the proposed approach, DAG is assessed on MuJoCo robotic control tasks and AntMaze tasks from the D4RL benchmark across datasets with varying data quality. Experimental results demonstrate that DAG effectively exploits advantage information and outperforms existing offline reinforcement learning approaches across multiple tasks. Compared with the TD3+BC baseline, DAG improves the total normalized score by 9.69% on MuJoCo and 10.06% on AntMaze.
Hui-Zhi Wang, Yan Kong· International journal of sof...· 0 citations
Offline reinforcement learning (RL) must reconcile two competing requirements: policy updates should stay near dataset-supported actions to keep value estimates reliable, yet meaningful gains often require moving beyond the behavior distribution. We develop a geometric view of offline actor updates by modeling policies as a probability manifold endowed with a chosen metric geometry. Under this lens, a broad class of offline actor objectives can be interpreted as a single proximal policy improvement step (SPI), i.e., an implicit discretization of a manifold gradient flow induced by a critic-defined energy. Building on this insight, we propose multi-step proximal policy improvement (MPI), a plug-in refinement mechanism that composes sequential re-centered proximal steps. MPI enables controlled policy improvement beyond dataset support while retaining proximal control at each refinement. The framework accommodates multiple policy geometries and admits practical instantiations for deterministic and diagonal-Gaussian policies. Experiments on D4RL benchmarks show that small numbers of MPI refinements improve strong offline baselines, including TD3+BC, ReBRAC, and IQL, on many tasks. Focused diagnostics further distinguish re-centered refinement from fixed-objective update scheduling and characterize limitations under critic error.
EvoHIL is presented, a unified framework that adapts the reward model, action generator, and visual do main within a staged human-in-the-loop learning process to improve task success, agreement with human-confirmation labels, motion smoothness, and completion time relative to human-in-the-loop and imitation baselines.
Shuoqing Zhang, Tongtong Cheng, Xiru Gao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
