This work proposes CoDrift, a compositional framework for one-step generative policy learning that combines three objective-level fields into a unified policy field that compares favorably with state-of-the-art methods and achieves the best average rank in both settings.
Abstract
Offline reinforcement learning is intrinsically multi-objective: a policy must remain compatible with the behavioral support of a fixed dataset while preferentially selecting high-value actions. We recast these objectives in a common form by viewing each as an action-space motion field that specifies how generated actions should move. This perspective enables heterogeneous learning objectives to be combined directly through field composition. Inspired by drifting models, we propose CoDrift, a compositional framework for one-step generative policy learning. CoDrift combines three objective-level fields into a unified policy field. The conditional field preserves state-dependent behavioral structure, while the marginal field pools actions across states to provide a more stable generative signal in the single-positive-sample regime of continuous-control offline RL. The value field moves generated actions toward higher-value regions. The composed field is absorbed into a stochastic generator that produces an action with a single forward pass at deployment. We evaluate CoDrift on 73 tasks from OGBench and D4RL in both offline and offline-to-online settings. CoDrift compares favorably with state-of-the-art methods and achieves the best average rank in both settings.
This work revisits where capacity should be invested in an offline actor--critic method and proposes LAC (Light Actor, deep Critic), a lightweight deterministic actor that matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency.
ReBRAC-v2 is introduced, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection, and ranks first in eight categories.
Offline reinforcement learning (RL) aims to derive effective policies from fixed datasets without environment interaction. While generative models such as diffusion and flow matching improve policy expressiveness, they often suffer from high computational cost due to multi-step sampling and limited representational richness from uninformative Gaussian priors. To address these challenges, we propose Latent MeanFlow Policy Optimization (LaMPO), a generative policy framework that formulates offline RL as latent generative policy optimization. LaMPO learns a behavior-conditioned latent distribution, which provides an informative prior and reduces the modeling burden of the generative process. Conditioned on this prior, the MeanFlow policy is trained to predict the average action velocity, enabling high-fidelity one-step action generation that avoids the cost of iterative sampling. Guided by the MeanFlow policy, the target policy is then optimized under a MeanFlow-induced behavior constraint, achieving effective policy improvement. We further establish a theoretical performance lower bound of the target policy relative to the MeanFlow policy. Extensive evaluations across 71 tasks in OGBench, D4RL, and real-world humanoid manipulation demonstrate that LaMPO achieves superior performance with high efficiency, yielding a 19% average improvement over existing state-of-the-art methods and an 81% success rate in real-world robotic tasks.
Tenglong Liu, Xin Xu, Yixing Lan et al.· IEEE Transactions on Pattern...· 0 citations
Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through the singular structure of model weights and identify spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. We operationalize spectral inheritance as Isospectral Optimization (ISO), an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations. Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model, requiring no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD). It recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods. Online, ISO-Optimizer applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed. Across reasoning and coding tasks ranging from 1.5B to 8B parameters, ISO-Optimizer improves accuracy in the reported runs and reaches matched scores with substantially fewer training steps. On Qwen3-8B-Base, AdamW reaches an aggregate accuracy of 0.495 after 270 training steps. ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 after 210 training steps. Together, ISO offers a concrete answer to RLVR's missing optimization layer: rather than inheriting pre-training optimization wholesale, design post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.
The key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement, which provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors.
Michael Murray, Daphne Chen, Simran Bagaria et al.· 1 citation
VINE is proposed, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies and achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task.
Rushuai Yang, Zhuo Han, Houlin Li et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
