Skip to content
Conference

Multimodal data-driven knowledge graph reasoning model for historical and cultural resources

Aug 2026 · International Conference on Advanced Manufacturing, Automation, and Deep Learning · Vol 14311, pp. 143110G - 143110G-7 · 0 citations · 10 references
Engineering

TL;DR

A multi-modal data-driven knowledge graph reasoning model integrating unified encoding, graph representation learning, and rule constraints is proposed, indicating better relation completion accuracy, reasoning stability, and inference efficiency in multi-modal knowledge graph reasoning tasks.

Abstract

To address the problems of heterogeneous structures, weak semantic alignment, sparse relations, and reasoning conflicts in multi-modal knowledge graph construction, this paper proposes a multi-modal data-driven knowledge graph reasoning model integrating unified encoding, graph representation learning, and rule constraints. Text descriptions, image labels, temporal attributes, and spatial coordinates are mapped into a shared representation space through modality-specific encoders and feature projection matrices. A multi-relational knowledge graph is then constructed through entity extraction, relation identification, triplet generation, and graph storage. On this basis, graph representation learning is introduced to aggregate neighborhood information and mine implicit relations among sparse nodes. To reduce semantic drift and invalid link prediction, temporal order, spatial inclusion, entity dependency, and event consistency rules are embedded into the reasoning score function. Candidate filtering and conflict resolution are further designed to improve inference efficiency under large-scale triplet conditions. Experimental results show that the proposed model achieves 0.894 Precision, 0.872 Recall, 0.883 F1, 0.907 MRR, and 0.931 Hits@10, outperforming TransE, GCN, GAT, R-GCN, and CompGCN. When the candidate triplet scale reaches 10,000, the inference response time of the proposed model is 146 ms, indicating better relation completion accuracy, reasoning stability, and inference efficiency in multi-modal knowledge graph reasoning tasks.

View source

Similar papers

Preprint Aug 2026

ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

ViSR-KGC, a visual subgraph reasoning approach for KGC, integrates three complementary capabilities to capture semantic correlations: identifying global topology dependencies via representation learning, analyzing local multimodal evidence using VLMs, and providing necessary commonsense knowledge inherent in pre-trained models.

Jiafan Li, Mengxue Yang, Jiaqi Zhu et al. · 0 citations
Open access Sep 2026

A study on the GraphRAG semantic retrieval algorithm for multimodal data

To address the challenges of evidence chain breakage of vectors, collaborative constraint between image and structured fields is challenging, and the credibility of the generated results is insufficient with multimodal data, this paper proposes a GraphRAG semantic retrieval model for multimodal data. In order to realize the associative expression of text, image and structured data in the same retrieval link, based on the traditional RAG and knowledge graph enhanced retrieval, the model constructs a five-layer framework of data ingestion–semantic encoding–graph construction–retrieval enhancement–controlled generation. Text paragraphs, image target regions, and structured fields are encoded into modality-specific representations and subsequently projected into a unified 640-dimensional semantic retrieval space. Entity extraction, cross-modal entity linking, graph-path expansion, evidence aggregation, and credibility-aware re-ranking are then performed in this shared space to construct traceable multimodal evidence chains. A total of 18,640 technical texts and 7820 device images, 42,300 table records and 3600 retrieval questions were selected for verification in the experiment. The results showed that after GraphRAG enhancement, the median of Precision@5 increased from 0.812 to 0.869, the median of Recall @ 10 increased from 0.835 to 0.895, and the evidence hit rate increased from 84.1 to 89.7%. The Precision@5 and Recall @ 10 of the three-modal joint retrieval were 0.904 and 0.928, respectively, while the average end-to-end query time was 283 ms. Results demonstrate the ability of the model to increase the retrieval accuracy, evidence integrity and generation credibility in complex multimodal scenarios.

Chun-Jing Liao, Pei-Shan Ye, An-Ni Huang et al. · 0 citations
Open access Jul 2026

Multi-curvature progressive fusion for knowledge graph completion

Knowledge graphs have become a fundamental representation for structured knowledge, yet their incompleteness remains a major obstacle to reliable reasoning. Missing links may interrupt relational paths and limit the evidence available for downstream inference. Knowledge graph completion addresses this problem by predicting missing facts from observed relational patterns. Existing methods have made notable progress, but many of them rely on a single geometric space or fuse multiple geometric representations in a static manner. This limits their ability to capture heterogeneous graph structures, especially when hierarchical dependencies, local relational patterns, and cyclic motifs coexist in the same knowledge graph. In this paper, we propose multi-curvature progressive fusion (MCPF), a geometry-aware graph neural reasoning framework for knowledge graph completion. MCPF learns relation-aware representations in Euclidean, hyperbolic, and hyperspherical spaces, which provide complementary inductive biases for conventional relational interactions, hierarchical organization, and cyclic structures. Instead of directly aggregating these representations, MCPF progressively refines them through cross-space interactions and uses a query-conditioned gating mechanism to determine their relative contributions. This design enables the model to emphasize reliable geometric signals while suppressing less informative ones for each query. The fused representation is then used within a graph neural reasoning architecture for message passing and link prediction. Experiments on four benchmark datasets demonstrate that MCPF achieves competitive results against representative baselines and consistently outperforms its ablated variants.

Muhua Dang, Xinde Yu, Zhao Jin · 0 citations
Open access Aug 2026

Temporal-spatial reasoning over hypergraph knowledge structures for multimodal retrieval-augmented generation

Existing graph-based retrieval-augmented generation (RAG) systems represent knowledge with binary relations and rely primarily on semantic similarity for retrieval. This design struggles with multimodal queries requiring temporal constraints, spatial relationships, or higher-order interactions among entities. We present HyperTSRAG, a multimodal RAG retrieval algorithm that models knowledge as a hypergraph and performs explicit temporal-spatial reasoning during traversal. HyperTSRAG performs a bipartite-alternating best-first traversal that alternates between entity nodes and hyperedge nodes to capture higher-order connectivity and ranks candidate evidence using a scoring function applied within traversal that integrates semantic similarity, temporal coherence, spatial overlap, and structural importance. We evaluate HyperTSRAG on a benchmark corpus of 1,000 multimodal documents (text, images, audio, and video) with 500 queries spanning simple lookups through complex multi-hop reasoning. For complex queries, HyperTSRAG achieves 78.3% Recall@10, improving by 12.3% over GraphRAG and 18.3% over LightRAG, while maintaining a 95th-percentile latency of 1.83 s. On temporal-spatial subsets, HyperTSRAG attains 85.2% accuracy on temporal queries and 81.6% on spatial queries. Ablation studies show that individual scoring components contribute 2–5% gains. Relative to a matched Projected-Binary-RAG control derived from the same extracted hyperedges, HyperTSRAG improves Recall@10 by 9.1%, while a separate binary-graph traversal ablation yields a 13.4% decrease, underscoring the value of native n-ary structure. These results support hypergraph-native traversal with temporal-spatial-aware scoring as an effective retrieval paradigm for the evaluated multimodal RAG setting, particularly for constraint-driven queries that semantic-only retrieval cannot address.

Timothy Dillan, S. Isa, A. S. Girsang et al. · 0 citations
Open access Aug 2026

Development of Knowledge Graph Construction and Intelligent Question Answering System in Education Based on Transformer Model

Efficient semantic information processing and multi-hop knowledge reasoning have become essential technologies for intelligent information services and next-generation networked systems. To address inaccurate semantic understanding caused by short or ambiguous queries and insufficient reasoning capability under fragmented knowledge structures, this study proposes an intelligent question answering framework that tightly integrates Transformer-based semantic encoding with graph attention reasoning. The proposed architecture employs DeBERTa-v3-base and conditional random fields for entity recognition, combines dual-tower vector retrieval with cross-encoder reranking for semantic disambiguation, and constructs k-hop knowledge subgraphs enhanced by multi-layer Graph Attention Networks to achieve relation-aware information propagation and evidence aggregation. A multi-task joint optimization strategy incorporating adaptive gradient normalization, adjacency reconstruction regularization, and negative sampling is further introduced to improve long-tail generalization and reasoning robustness. Neo4j graph storage and FAISS vector indexing enable near real-time retrieval and scalable deployment. Experimental evaluation demonstrates high semantic understanding accuracy, interpretable multi-hop reasoning capability, and stable performance under short and colloquial queries, with superior Exact Match and path reasoning accuracy compared with baseline models. Beyond educational applications, the proposed framework provides an effective methodology for semantic information fusion, distributed knowledge reasoning, intelligent query processing, and adaptive decision support, offering valuable references for communication-enabled information systems, networked knowledge services, and intelligent information infrastructures related to Electromagnetic Waves, Antennas and Propagation engineering applications.

Zi-Xian Bai, Q. Mu · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.