Skip to content

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

Sep 2026 · 0 citations · 46 references
Computer Science

TL;DR

EyeVQA is introduced, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs, and fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol are benchmarked.

Abstract

Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized. Existing ophthalmic datasets are typically designed for individual diseases or specialized tasks, making it difficult to systematically evaluate whether VLMs can move beyond disease recognition toward comparative reasoning and fine-grained spatial grounding. We introduce EyeVQA, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs. EyeVQA is constructed from 21 available ophthalmic datasets and contains 20,000 question-answer pairs spanning six disease groups and seven question types: Single-Choice, Multi-Select, Variable-Select, True-False, Ranking, Point Location, and Bounding Box. Gold answers are deterministically derived from source-provided diagnoses, severity grades, clinical findings, segmentation masks, bounding boxes, and anatomical landmarks, enabling reproducible evaluation without relying on model-generated annotations. Notably, 44.5% of the questions require reasoning across multiple images, extending evaluation beyond conventional single-image medical VQA. We benchmark fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol. The best-performing model only achieves an overall score of 62.8, while substantial gaps remain in spatial grounding and cross-task generalization. These results highlight the limitations of current VLMs in comprehensive ophthalmic visual understanding and establish EyeVQA as a diagnostic benchmark for developing more reliable and spatially grounded ophthalmic multimodal models. The project page is available at https://github.com/PKUTHM/EyeVQA.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards and demonstrates that retinal model evaluation is inherently multidimensional.

David S. Restrepo, Chen-Wei Wu, L. Nakayama et al. · 0 citations
Open access Sep 2026

An Automated, Contamination-Controlled VQA Benchmark for Evaluating Vision-Language Models on 3D Oncology Imaging

Vision-language models (VLMs) are increasingly applied to medical imaging, yet public benchmarks may reward memorization over perception: their images and questions can enter pretraining corpora, and many items remain answerable from question text alone. We present an automated, agent-driven pipeline that builds multip...

Bo Liu, Han Gu, Xiang-Rui Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation

A clinically curated Pan-Asia WSI--report dataset is introduced and the REG 2025 benchmark is established as a benchmark for evaluating WSI-based structured report generation and vision-language understanding in computational pathology, providing insights for the design of clinically grounded multimodal pathology model...

Yu-Mi Lee, Harim Oh, Hyo-yun Kim et al. · 0 citations

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

A mechanistic analysis of the VLM model's internal attention patterns shows that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation, and task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in t...

Janet Wang, Yun-Bei Zhang, Xiao Wang et al. · 0 citations
#small language model Review Open access Sep 2026

Large language models for ophthalmic examination understanding: from information extraction to clinical decision support

Publicly available LLMs and MLLMs are evolving from report-parsing tools toward broader ophthalmic clinical assistants, but their use should follow a task-layered validation framework in which verification and human oversight increase with the clinical consequences of error.

Gang-Yi Wang, Xuan-Qiao Lin, Yi-Zhou Yang · 0 citations
Preprint Aug 2026

Beyond Natural-Image Foundation Models: Benchmarking Satellite Pretraining for Ophthalmic Image Analysis

Satellite imagery is proposed as a novel pretraining domain for MedVFM development and benchmarking, motivated by its closer visual alignment with medical data and its freedom from the privacy constraints that limit medical datasets.

Lovre Antonio Budimir, Ming Gong, Alyssa Foong Quinney et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.