Multimodal Knowledge Graph Link Prediction Model Based on Sequential Fusion

Abstract

In the research of multimodal knowledge graph link prediction, the existing methods usually use concatenation or attention mechanism to fuse multimodal features. However, these methods fail to fully solve the problem of heterogeneity between different modalities and their complex interactions, which has a certain impact on the accuracy of link prediction. Therefore, in order to solve this problem, this paper proposes a multi-modal knowledge graph link prediction model based on sequential fusion, which uses a sequential fusion method to gradually fuse multi-modal information. In this method, a two-level fusion module is designed to fuse visual information and text information at two levels of coarse granularity and fine granularity, and then the orthogonal fusion module is used to integrate the structural information into the existing visual text features to ensure the synergy between different modalities. Moreover, regularization is added to the loss function to reduce redundant features and improve the performance of link prediction. In order to verify the effectiveness of the proposed method, experiments are carried out on FB15k-237-IMG and WN18-IMG datasets, and compared with the existing multi-modal models, the proposed model achieves advanced results.