Skip to content
Preprint

An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography

Aug 2026 · 0 citations · 45 references
Computer Science

TL;DR

An agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography was developed and validated, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.

Abstract

Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency. We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography. The workflow had three steps: (1) LLM initial assessment; (2) function calling to invoke specialized tools for image quality (QAModel, FundaQ-8), glaucoma classification (SwinV2-Tiny), and optic disc/cup segmentation (SegFormer-B0); and (3) LLM reflection integrating the initial impression with tool outputs. Two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) were evaluated on two public datasets (ORIGA, n=100; RIM-ONE-v3, n=100) under uncropped and cropped fields of view; all images were independently graded by a masked fellowship-trained glaucoma specialist. The agentic workflow improved classification accuracy by 16 to 47 percentage points across all conditions, reaching within 6 points of the specialist; on RIM-ONE-v3 the best configurations matched the specialist accuracy of 88%. LLM-alone approaches failed in two ways: GPT-5.4 mini showed positive bias (sensitivity 95-100%, specificity 0-5%), while Gemini 2.5 Flash varied stochastically between runs; the agentic workflow corrected both. Cup-to-disc ratio error fell 15-50% (MAE 0.156-0.228 to 0.104-0.132), and correlation with specialist grading rose from weak (r=0.12-0.39) to moderate-strong (r=0.59-0.84). Run-to-run consistency rose from near-random (kappa as low as -0.01) to near-perfect (kappa up to 0.96). Integrating LLMs with specialized tools addressed key limitations of LLM-alone approaches, including over-diagnosis and run-to-run variability. Gains held for both LLMs, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards and demonstrates that retinal model evaluation is inherently multidimensional.

David S. Restrepo, Chen-Wei Wu, L. Nakayama et al. · 0 citations
Conference Aug 2026

Efficientnet-B0 and Vlm-Based Explainable Framework for Diabetic Retinopathy Detection and Severity Grading

Diabetes mellitus (DM) is one of the most common causes of preventable blindness in the world, and DR is a major cause of working-age adults losing their vision. Mass screening programmes are necessary but limited by the availability of trained ophthalmologists, and the cost of such programmes. This paper introduces a...

Varsha Pandagre, N. Deshpande, Shilpa Gite · 0 citations
Aug 2026

Diagnostic Accuracy of Multimodal Large Language Models in Retinal Fundus Photography.

ChatGPT demonstrated the strongest accuracy, justification-accuracy association, and confidence-accuracy correlation compared with Claude, Gemini, and Grok and showed a positive correlation between confidence and accuracy.

Lia Huo, Astha Chandra, Michael Balas et al. · 0 citations
Open access Sep 2026

Comparative Evaluation of EfficientNet-B3, Vision Transformers, and RETFound for Seven-Category Retinopathy Fundus Image Classification

Background/Objectives: Retinal fundus photography is the most widely available means of screening for sight-threatening disease, but it is difficult to read: several conditions leave overlapping signs on the same photograph, the features that separate them are often subtle and confined to small regions of the retina, a...

Majid Alutaibi, Meshari Alazmi · 0 citations
Open access Oct 2026

RFM: A Lightweight Retinal Foundation Model for Generalised Oculomics

Foundation models for colour fundus photography (CFP) are candidate visual encoders for multimodal medical AI, but existing models are large and are assessed under fine-tuning rather than the frozen-encoder conditions such systems impose. We present RFM (Retinal Foundation Model), a 64-million-parameter Vision Transfor...

J. Fajtl, R. Welikala, R. Shakespeare et al. · 0 citations
Open access Sep 2026

A multi-label deep learning system for simultaneous detection of nine fundus conditions.

PURPOSE To develop and validate a deep learning model for simultaneous detection of nine fundus conditions from a single color fundus photograph. METHODS A development dataset of 236,242 color fundus images was assembled from 17 heterogeneous sources and partitioned at the patient level into training (70%), tuning (1...

Jungwoo Ha, Ye-On-Hee Choi, Dan-Bi Lee et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.