Semantic Disambiguation Transformer for Multimodal Emotion Understanding
Abstract
The fast evolution of social media platforms, multimodal content combining textual and visual information has become a foremost part for expressing emotions and opinions. Precisely understanding such content is challenging due to the presence of implicit semantics, contextual ambiguity, and multimodal relations. A Semantic Disambiguation Transformer for fine-grained multimodal emotion analysis, which integrates contextual semantic learning with multimodal feature fusion is proposed. The framework employs a BERT-based encoder for textual representation and a ResNet-50 architecture for visual feature extraction. To improve semantic understanding, a word sense disambiguation mechanism is combined through pretraining on SemEval datasets. This fusion approach is used to align multimodal features into a shared embedding space, followed by a multi-task learning framework for predicting humour, sarcasm, offensiveness, motivation, sentiment, and hate. The model is assessed using accuracy, macro F1-score, and AUC, supported by visualization techniques such as confusion matrices, ROC curves, precision recall curves, and semantic heatmaps. Experimental results demonstrate the effectiveness of the proposed approach in capturing multimodal semantics and improving emotion classification performance.