Traj-MCPO: Trajectory-wise Marginal Consistent Policy Optimization
Abstract
Rectified flow models are increasingly used for text-to-image generation, a core multimedia content generation task, thanks to efficient, near-deterministic ODE sampling, but aligning them with human preferences or prompt-following objectives still commonly relies on reinforcement learning with stochastic rollouts. Existing extensions of GRPO to flow-based generation inject noise into the reverse dynamics and optimize terminal rewards with step-wise clipped surrogates, which introduces two key mismatches: an ODE-SDE terminal marginal mismatch between training and inference, and a reward-granularity mismatch caused by repeatedly reweighting a single terminal reward across long denoising trajectories. To address this, we propose Traj-MCPO, a trajectory-wise policy optimization framework for rectified flows under terminal rewards. Starting from the marginal terminal objective, we derive an equivalent trajectory-form policy gradient and build a trajectory-wise clipped surrogate with group-relative advantages. At the core of Traj-MCPO is a length-normalized trajectory likelihood ratio, which aggregates step-wise likelihood changes into a single horizon-normalized quantity so that all timesteps contribute uniformly through an averaged log-likelihood gradient. Building on this formulation, trajectory-wise clipping directly constrains the average log-likelihood shift between the updated and behavior policies, thereby reducing rollout drift and improving consistency with inference-time ODE sampling. Experiments on text-to-image post-training with reward models show that Traj-MCPO stabilizes optimization and consistently improves prompt following and image quality over step-wise GRPO baselines.