Conference Open access

Omni-RewardBench: Toward a Comprehensive Evaluation of Generative Reward Models Across Modalities

Chi-Min Chan Yujin Zhou Pengcheng Wen Bo Yin Jiaming Ji Juntao Dai Wei Xue Sirui Han Yike Guo

2026 · Annual Meeting of the Association for Computational Linguistics · pp. 13962-13984 · 0 citations · 49 references

Computer Science

TL;DR

Omni-RewardBench is introduced, the first benchmark for comprehensive evaluation of ORMs across modalities and demonstrates that current OLLMs fall short as reward models, revealing several common failure modes such as perception failure, modality dominance failure, and cross-modal fusion failure.

Abstract

The rise of Omni-modality Large Language Models (OLLMs) capable of jointly processing text, audio, and visual inputs marks a major step toward general intelligence. Ensuring their alignment with human preferences requires effective Omni-modality Reward Models (ORMs), which serve as surrogates for human judgment to guide OLLMs behavior. However, ORMs evaluation remains under-developed in the previous literature. Existing benchmarks are largely text-centric or limited to bimodal tasks, restricting comprehensive assessment for ORMs. To bridge this gap, we introduce Omni-RewardBench , the first benchmark for comprehensive evaluation of ORMs across modalities. In short, our contributions are threefold: (1) a hybrid automatic-annotation and human-verification pipeline to construct high-quality evaluation data; (2) extensive experiments on 20+ models, including inherently omni-modal and modality-bridged systems. Our experimental results demonstrate that current OLLMs fall short as reward models, revealing several common failure modes such as perception failure , modality dominance failure , and cross-modal fusion failure . and (3) strong correlations between Omni-RewardBench scores and downstream performance (IID r = 0.94, OOD r = 0.72), validating its reliability as a predictor of real-world capability and alignment quality.

DOI Semantic Scholar

Read PDF

Similar papers

Preprint Aug 2026

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning

Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, l...

Pyrros Koussios, Chen-Hao Li, Xin Chen et al. · 0 citations

VisPlay : Self-Evolving Vision-Language Models

VisPlay is introduced, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning capabilities from massive unlabeled image data and establishes a scalable path toward self-evolving multimodal intelligence.

Yicheng He, Chengsong Huang, Zongxia Li et al. · 0 citations

VEG: Verbal ϵ -greedy for Semantic Exploration in Multi-Turn RL Agents

This work proposes VEG (verbal ϵ -greedy), a novel framework that leverages external feedback as a dynamic control variable to explicitly balance exploration and exploitation within the semantic space and achieves superior accuracy compared to standard RL baselines.

Yongchang Hao, Jie Hao, Yongsheng Mei et al. · 0 citations

Preprint Aug 2026

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

It is shown that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines and demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinfor...

Srivalli Katkuri, Maxwell Kawada, J. Wachs · 0 citations

Preprint Aug 2026

Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

Evaluation-Conditioned Training (ECT), a post-training framework that uses natural language to condition each training sample on the fidelity of the feedback the authors provide and then elicits the desired behavior by conditioning the LLM on a high-fidelity monitor in deployment, is introduced.

Alec Harris, Kasey Corra, Archie Chaudhury et al. · 0 citations

Review Open access Aug 2026

Reinforcement Learning for Multimodal Foundation Models: A Survey

A coherent map of the rapidly expanding landscape of visual RL is provided to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry.

Weijia Wu, Chen Gao, Joya Chen et al. · ACM Computing Surveys · 0 citations