Intelligent Prompt Construction for Large Language Models in Knowledge-based Visual Question Answering

Generating then Refining for Reliable Knowledge Base Question Answering

Evaluations on standard KBQA benchmarks show that the proposed ARI-KBQA enhances model performance with a reduced search space, especially in complex multi-hop query scenarios.

Structured multi-level knowledge augmentation via small-to-large evidence-guided collaboration for knowledge-based VQA

Knowledge-based visual question answering requires models to connect question-relevant visual evidence with external knowledge for accurate reasoning. However, existing approaches still face two critical challenges: insufficient alignment between visual content and question intent, which leads to missing or irrelevant evidence, and ambiguity in entity-level semantics, especially for fine-grained or knowledge-intensive concepts. To address these issues, we propose an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA. The framework uses lightweight vision–language models to construct structured textual evidence, which is subsequently provided to a frozen large language model (LLM) for final reasoning. We emphasize that the proposed method does not introduce a new LLM architecture or a new training mechanism; instead, it focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference. The framework consists of four complementary evidence-construction modules: (1) a question-oriented image information extraction module that generates query-relevant visual descriptions to enhance visual–semantic alignment; (2) an entity enhancement module that introduces clarifying sub-questions to alleviate entity-level ambiguity; (3) a candidate-guided answer generation module that provides plausible answer cues to constrain the reasoning space; and (4) a contextual exemplar retrieval module that supplies relevant demonstrations to support knowledge-grounded inference. Experiments on OK-VQA and A-OKVQA show that the proposed method achieves 66.72% and 69.51% accuracy, respectively, outperforming strong baselines, while supplementary analyses examine its robustness, output-format reliability, and inference cost.

A visual question answering model based on entity knowledge selection

Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning

Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.

AutoDavis: Automatic and Dynamic Evaluation Protocol of Large Vision-Language Models on Visual Question-Answering

Large Vision-Language Models (LVLMs) have become essential for advancing the integration of visual and linguistic information. While existing benchmarks have laid a solid foundation for evaluation, they are often static, resource-intensive to build, and limited in adaptability. In comparison, automatic evaluation has shown promise in the textual domain, but the visual modality remains far less explored. To advance this frontier, in this work, we introduce AutoDavis, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions. AutoDavis leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. To ensure data diversity, our framework employs a hierarchical aspect-driven generation process enhanced with semantic graph-based constraints. To safeguard reliability, the framework incorporates a self-validation mechanism to detect and correct errors, along with an error-driven adjustment module to mitigate potential bias. Through an extensive evaluation of 11 popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence. View website for code and data.

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: https://github.com/MINE-USTC/SAFE-G.