Analysis of Multimodal Heterogeneous Graph Transformers for Reconstructing Social Network Structures Based on Open-Source Data
In this paper, transformer-based graph neural networks are applied across different data types, such as text, images, and metadata, to automatically infer the structure of social networks from publicly available OSINT data. A pipeline was built that first processes text with BERT and images with ResNet, then combines them with an attention mechanism, and finally feeds the fused representations into a Heterogeneous Graph Transformer to classify nodes and predict links between them. Experiments were conducted on two publicly available Twitter datasets for research: Cresci-15 and Twibot-20. For node classification, our method achieved F1-scores of 0.979 and 0.882 on these datasets, which are better than using only text or only the graph structure. For link prediction, we got an AUC of 0.871 on Twibot-20. Combining different types of data with a heterogeneous graph structure performs better than using a single type, but our method has certain limitations that warrant future work.