Preprint

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Yun-Heng Li Guo-Hong Mu Hao Li Sheng-Sheng Qian Ding-Wen Zhang Qi-Bin Hou Ming-Ming Cheng

Aug 2026 · 1 citation · 115 references

Computer Science

TL;DR

The sample efficiency and scalability of RL post-training for video MLLMs and introduces OraRL, a decoupled advantage estimator that scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts.

Abstract

Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

arXiv Semantic Scholar

View source

Similar papers

Efficient Frame Selection for Long Video Understanding via Reinforcement Learning

A lightweight frame selection method to identify keyframes and train it via a two-stage strategy that improves performance of diverse down-stream MLLMs across benchmarks spanning medium to long videos.

Yaxuan Qin, Hefei Li, Wenqi Mu et al. · 2 citations

Preprint Aug 2026

Improving Generalization Robustness of Multimodal RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two iss...

P. Zhou, Zhiwei Tang, Xiaopeng Peng et al. · 0 citations

Jul 2026

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

HiF4 is established as the enabling format for end-to-end FP4 RL post-training, and Rollout Residual Quantization (Rollout-ResQ) is established as the activation-side mechanism that makes the gap to BF16 closable.

H. Mak, Shadan Golestan, H. Le et al. · arXiv.org · 0 citations

Jul 2026

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

This work identifies activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid.

Zhengyang Zhuge, Hao Yu, Xin Wang et al. · arXiv.org · 0 citations

Preprint Aug 2026

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Noise-Contrastive GRPO is introduced, which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

Michael M. Jerge, Joseph Pelczar, J. Downes · 1 citation

Preprint Jul 2026

SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness, is introduced, establishing itself as a robust, model-agnostic solution for prompt learning.

Hossein Rajoli, Fatemeh Lotfi, Niloufar Alipour Talemi et al. · 0 citations