TRAG-DPO: Enhancing Visual Question Answering in Tomato Disease Diagnosis via Hybrid Retrieval and DPO-Based Preference Alignment

Abstract

Advances in intelligent vision systems have advanced tomato disease diagnosis from manual inspection to automated image-based analysis. Most prior deep learning approaches formulate tomato disease diagnosis as closed-set image classification, producing only coarse disease labels without interpretability or interactive reasoning. These limitations motivate visual question answering based on vision-language models, which provide question-driven explanations and guidance. However, existing models often struggle to perceive small and subtle lesion regions, suffer from cross-modal mismatch when retrieving external agronomic knowledge, and generate responses that are not well aligned with diagnostic intent. To overcome these challenges, we propose TRAG-DPO, a unified retrieval-augmented vision-language framework for tomato disease visual question answering that formulates diagnosis as a knowledge-guided interactive reasoning process. The framework comprises three modules: 1) A fine-grained lesion-aware visual encoding (FG-ViT) that enhances sensitivity to localized lesion regions while preserving global structural context. 2) A coarse-to-fine hybrid retrieval mechanism that narrows the semantic search space through image-text category matching and then selects question-relevant passages to provide evidence-based context. 3) A preference alignment mechanism that employs Direct Preference Optimization to transform retrieved agronomic knowledge into concise and intent-aligned responses. We further construct a composite resource integrating a tomato-specific visual instruction dataset, a disease knowledge base, and a preference fine-tuning dataset with expert-validated comparisons. Extensive experiments on both closed-ended and open-ended tomato VQA benchmarks demonstrate that TRAG-DPO consistently outperforms both general-purpose and agriculture-specific vision-language models. In addition, cross-crop experiments on corn and rice disease VQA further verify the transferability and scalability of the proposed framework. Data and code are available at: https://github.com/SaaSttt/TRAG-DPO