SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

Cheng Tang Junzhi Ning Min Cen Wei Li Xin-Yi Zeng Pinxian Zeng Rongbin Li Qi-Ming Zhu Yuqiang Li Junjun He Yirong Chen Ming Hu

Jul 2026 · arXiv.org · Vol abs/2607.13931 · 0 citations · 44 references

Computer Science

TL;DR

SIVA-RL is proposed, a Sensitivity-Invariance Visual Alignment framework that replaces operator-conditioned regularization with sample-wise, outcome-conditioned supervision and yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.

Abstract

Reinforcement learning with verifiable rewards (RLVR) drives multimodal reasoning, but answer-level correctness does not guarantee that a vision-language model grounds its predictions in visual evidence. Existing visual-intervention methods contrast policy behavior on original and modified images, yet assign supervision by the type of intervention rather than its observed effect. This assumption fails: identical operators produce heterogeneous outcomes across samples. We propose SIVA-RL, a Sensitivity-Invariance Visual Alignment framework that replaces operator-conditioned regularization with sample-wise, outcome-conditioned supervision. SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image PatchSwap. A frozen audit policy then scores each clean-intervention pair, and the observed reward drop becomes soft routing weights. Large-drop pairs drive sensitivity alignment, low-drop pairs drive clean-anchored invariance alignment, and ambiguous pairs are down-weighted. This design decouples intervention construction from supervision assignment and is compatible with both GRPO and DAPO backbones. Across nine multimodal reasoning benchmarks spanning mathematical, logical, and vision-dependent tasks, SIVA-RL improves 3B and 7B models over matched RL baselines in every setting. It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.

DOI arXiv Semantic Scholar

View source

Similar papers

Review Open access Aug 2026

Reinforcement Learning for Multimodal Foundation Models: A Survey

A coherent map of the rapidly expanding landscape of visual RL is provided to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry.

Weijia Wu, Chen Gao, Joya Chen et al. · ACM Computing Surveys · 0 citations

Jul 2026

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts, and refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction.

Binbin Zheng, Zi-Jun Xie, Guan-Qun Zhao et al. · arXiv.org · 1 citation

#machine learning Preprint Sep 2026

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.

Le-Qi Zheng, Jin-Bo Su, Fang Niu et al. · 2 citations

Jul 2026

H2SD: Hybrid Hindsight Self-Distillation

Experiments on challenging reasoning benchmarks show that H$^2$SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.

Qi Cai, Yi-Chuan Ma, Linyang Li et al. · arXiv.org · 2 citations

Jul 2026

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function across auxiliary tasks before RLHF training, is introduced, providing theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization.

Yu-An Chu · arXiv.org · 0 citations

Preprint Aug 2026

EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

EvoHIL is presented, a unified framework that adapts the reward model, action generator, and visual do main within a staged human-in-the-loop learning process to improve task success, agreement with human-confirmation labels, motion smoothness, and completion time relative to human-in-the-loop and imitation baselines.

Shuoqing Zhang, Tongtong Cheng, Xiru Gao et al. · 0 citations