AutoJudger : An Adaptive Evaluation Framework for Efficient Benchmarking of MLLMs

Xuanwen Ding Chengjun Pan Zejun Li Ji-Wen Zhang Siyuan Wang Zhongyu Wei Emanuele Tschannen Thomas Bugliarello Unterthiner Daniel Skanda Keysers Fangyu Koppula Adam Liu Alexey Grycner N. Gritsenko Manoj Houlsby Kumar Eli Bingham John Chen M. Jankowiak F. Obermeyer Neeraj Pradhan Theofanis Karaletsos R. Singh Paul A. Szerlip Paul Horsfall Li Cai Kilchan Choi Mark Hansen Lauren Harrell Zhe Chen Weiyun Wang Hao Tian Shenglong Ye Zhangwei Gao Erfei Cui Wenwen Tong Kongzhi Hu Jia-Peng Luo Zheng Ma Jiannan Wu Wenhai Wang Weijie J. Su Guo Chen Sen Xing Muyan Zhong Qinglong Zhang Matt Deitke Christopher H. Clark Sangho Lee Rohun Tripathi Yue Yang Jae Sung Park Mohammadreza Salehi Niklas Muennighoff Kyle Lo Luca Soldaini Jiasen Lu Taira Anderson Erin Bransom Kiana Ehsani Huong Ngo YenSung Chen Ajay Patel Mark Yatskar Mucong Ding Chenghao Deng Jocelyn Choo Zichu Wu Aakriti Agrawal Avi Schwarzschild Tianyi Zhou Tom Goldstein John Langford Anima Anandkumar Khang T. Doan B. Huynh D. T. Hoang T.-H. Pham Quan Nhat H. Pham T. M. Nguyen B. Q. Vo Suong N. Hoang Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang Xueyi Lin Jinrui Yang Xiawu Zheng S. Galib Shanshan Wang Guan Xu Pascal Pfeiffer Ryan Chesler Mark Landry Sri Satish Wentao Ge Shunian Chen Hardy Chen Nuotian Chen Junying Chen Zhihong Chen Wenya Xie Shuo Yan Shuhao Gu Jialing Zhang Siyuan Zhou Kevin Yu Zhaohu Xing Liangdong Wang Zhou Cao Jintao Jia Zhuoyi Zhang Yixuan Wang Zhe Hu Bo-wen Zhang Jijie Li Dong Liang Yingli Zhao Yulong Ao

· 0 citations · 29 references

TL;DR

This work introduces A 2 -Judger, a novel MLLM-based A gentic instantiation of A uto Judger equipped with semantic-aware retrieval and dynamic memory that significantly improves sample efficiency while maintaining reliable evaluation results.

Semantic Scholar

View source

Similar papers

Book Open access Aug 2026

AutoDavis: Automatic and Dynamic Evaluation Protocol of Large Vision-Language Models on Visual Question-Answering

Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. While existing benchmarks have laid a solid foundation for evaluation, they are often static, resource-intensive to build, and limited in adaptability. In comparison, automatic evaluation has shown promise in the textual domain, but the visual modality remains far less explored. To advance this frontier, in this work, we introduce AutoDavis, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions. AutoDavis leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. To ensure data diversity, our framework employs a hierarchical aspect-driven generation process enhanced with semantic graph-based constraints. To safeguard reliability, the framework incorporates a self-validation mechanism to detect and correct errors, along with an error-driven adjustment module to mitigate potential bias. Through an extensive evaluation of 11 popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence. View website for code and data.

Han Bao, Yue Huang, Yanbo Wang et al. · Proceedings of the 32nd ACM... · 0 citations

#artificial intelligence Preprint Aug 2026

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs'proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models'error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive"consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.

Jinzhe Li, Geng-Xu Li, Jinnan Li et al. · 0 citations

#natural language process... Preprint Aug 2026

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

GenRubric is introduced, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution, and experiments show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.

Yifan Chen, Haitao Li, Qingyao Ai et al. · 0 citations

#machine learning Preprint Sep 2026

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech

Himil Vasava, Mingzhou Jiang · 0 citations

Preprint Aug 2026

LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

This work evaluates 10 state-of-the-art MLLMs and examines three factors that influence performance: reasoning patterns, auxiliary tools, and robustness to image perturbations, showing that MLLM accuracy decreases and varies substantially as computational complexity increases.

Ziyan Xiao, Yinghao Zhu, Wenting Zhang et al. · 0 citations

Preprint Aug 2026

Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.

Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil et al. · 0 citations