DualG-MRAG is proposed, a Dual-tier framework that introduces a decoupled architecture comprising Macro-reasoning and Micro-matching Graphs for Multimodal RAG to suppress retrieval noise by isolating global structural reasoning from fine-grained evidence matching, and introduces a dynamic programming decoding mechanism that extracts explicit reasoning paths directly from the GNN's forward pass.
Abstract
While Multimodal Retrieval-Augmented Generation (MM-RAG) has shown promising results, it still struggles with complex multi-hop reasoning tasks. Existing methods primarily focus on independent instance-level matching, which often fails to capture explicit relationships across modalities and documents. Although Graph-enhanced methods introduce structural modeling, they face a fundamental challenge in multimodal scenarios: incorporating fine-grained visual features leads to rapid graph expansion and retrieval noise, whereas coarse-grained representations cause the discarding of critical local evidence. To address this dilemma, we propose DualG-MRAG, a Dual-tier framework that introduces a decoupled architecture comprising Macro-reasoning and Micro-matching Graphs for Multimodal RAG. Specifically, to suppress retrieval noise by isolating global structural reasoning from fine-grained evidence matching, we construct a Macro Graph for global topological routing and a Micro Graph for precise local verification. Subsequently, to enable dynamic relevance propagation across heterogeneous evidence sources, we formulate retrieval as a query-driven message passing process via a GNN Retriever. Furthermore, to provide the generative model with coherent structural guidance, we introduce a dynamic programming decoding mechanism that extracts explicit reasoning paths directly from the GNN's forward pass, replacing the standard input of isolated document chunks. Extensive experiments demonstrate that DualG-MRAG outperforms baselines in both evidence recall and complex QA accuracy.
Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.
Jiacheng Cai, Zijin Hong, Zheng Yuan et al.· 0 citations
This paper formalizes the document structure as a Multimodal Hypergraph, utilizing hyperedges as unified semantic containers to encapsulate multi-way associations across text, images, and tables, thereby transcending point-to-point modeling and introducing an Anchor-driven Incremental Refinement mechanism.
Shenao Chen, Yidan Xu, Xiangmin Han et al.· 0 citations
Retrieval-Augmented Generation (RAG) has become a fundamental paradigm for enhancing Large Language Models (LLMs) with external knowledge. However, while recent structure-augmented approaches organize documents into graphs to improve information access, their retrieval strategies remain largely static, relying on similarity ranking or static probability diffusion. We identify that this paradigm suffers from two inherent limitations in complex reasoning: popularity bias, where retrieval paths are trapped by high-degree distractors, and signal decay, where relevance signals attenuate over long reasoning chains. To overcome these challenges, we propose NaviRAG, a novel framework that reformulates retrieval as a reinforcement learning-driven dynamic navigation problem on schema-less knowledge graphs (KGs). Unlike passive diffusion, NaviRAG employs an agent that actively traverses the graph to act as a search-space pruning engine, identifying logical multi-hop reasoning paths. Technically, we introduce three key components: (1) Structure-Aware Query Expansion, which bridges the modality gap between unstructured queries and structured graph seeds for precise initialization; (2) Target-Driven Reward Shaping, which provides dense supervision based on semantic progress toward gold documents, effectively mitigating the sparse reward problem in large-scale graph traversal; and (3) a Multi-View Hybrid Reranking strategy that operates on the highly-pruned candidate subgraph, integrating policy confidence, semantic relevance, and global structural importance to ensure robust candidate selection. Extensive experiments on three multi-hop QA datasets and two single-hop QA datasets demonstrate that NaviRAG significantly outperforms baselines, achieving state-of-the-art performance in multi-hop QA while maintaining robustness in single-hop QA. Our code and data are available at https://github.com/CkingEW/NaviRAG.
Jinghong Lei, Wang Kun, Zhigang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Retrieval-Augmented Generation (RAG) has established itself as a compelling strategy for grounding large language model outputs in documentary evidence. However, production deployments continue to rely almost exclusively on homogeneous text corpora, even as enterprise repositories grow increasingly heterogeneous blending technical schematics, radiological images, annotated diagrams, and unstructured prose within the same archival system. This mismatch between system design and data reality motivates the present work. We propose a Multimodal RAG framework that unifies text and image retrieval through four tightly coupled components: a dual-stream embedding engine, a learned four-class AI query router, an adaptive confidence threshold, and a session-aware context store. Text is encoded with the allmpnet-base-v2 Sentence Transformer; images are embedded in the same 768-dimensional space via a domain-adapted CLIP ViT-L/14 model fine-tuned on approximately 120,000 technical and clinical text-image pairs. Concatenating 768-d vectors yields a 1,536-d composite query that drives a single approximate nearest-neighbor (ANN) search simultaneously across both modalities. A fine-tuned DistilBERT router assigns each query to one of four retrieval pathways text-only, image-only, hybrid, or conversational at 91.3% accuracy. Experiments were run on three enterprise corpora (engineering manuals, clinical case summaries, and legal paperwork) with the results being a mean F1@5 of 0.90, an increase of 22% over a dense text-only baseline, and a reduction of 49% in the number of hallucinations. The median first-token latency is 1.34s, which meets the interactive-use target deployment-contexts. These results show that it is possible to implement modality-aware retrieval in a simple and realistic setting with real-world organizational constraints and that the results are reproducible.
E.Vijayakumar, G. A.· 2026 4th International Confe...· 0 citations
MCoRe, a multi-entry complementary retrieval framework with reflection-guided iteration for multi-hop QA that enables multi-entry complementary retrieval by indexing entry units at multiple semantic resolutions with explicit links to chunk evidence, and fusing cross-resolution hits via chunk-level voting to form a compact evidence set for answer generation.
Juxiang Zeng, Zhuohui Gao, Zhe Hou et al.· Proceedings of the 32nd ACM...· 0 citations
This study proposes STaR, a novel retriever fine-tuning framework that integrates BM25 similarity graph-based soft labeling with a triplet similarity learning strategy based on Sentence-BERT (SBERT), and introduces a triplet-aware SBERT training architecture that explicitly models relative semantic distances between queries and candidate passages, significantly enhancing retrieval ranking precision and semantic robustness.
Jiali Jiang, Chih-Yung Chang, Youxi Li et al.· Multimedia Systems· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
