MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples.
Abstract
Multimodal sentiment analysis (MSA) aims to predict sentiment polarity and intensity from heterogeneous inputs such as text, audio, and vision. While large language models (LLMs) offer strong semantic priors for MSA, effectively incorporating audio and visual signals effectively remains challenging. A key challenge is that audio and visual sentiment cues evolve over different temporal scales, yet many LLM-based methods compress these signals through shallow projection or coarse pooling before fusing them with text, which can weaken cross-modal alignment and erase fine-grained affective information. We propose MGSI, a multi-granularity sentiment integration framework for LLM-based MSA. MGSI first encodes audio and visual streams at short-, medium-, and long-range temporal scales, preserving both local variations and global affective trends. It then refines non-text features through text-guided alignment, and applies polarity- and intensity-aware enhancement to better handle ambiguous and near-neutral samples. The resulting multimodal representation is finally compressed into a small set of pseudo-tokens for efficient conditioning of a frozen LLM. Experiments on four public benchmarks show that MGSI substantially outperforms frozen-LLM baselines and remains competitive with strong multimodal methods. Further ablation and sensitivity analyses support the effectiveness of multi-granularity temporal modeling, text-guided refinement, and adaptive sentiment calibration.
Multimodal Sentiment Analysis (MSA) aims to integrate text, audio, and visual modalities to achieve accurate sentiment modeling. Existing methods often rely on shallow interaction structures, making it difficult to jointly capture fine-grained local dynamics and high-level semantic dependencies. In addition, dominant modalities may suppress the learning of weaker modalities, leading to insufficient cross-modal semantic alignment. To address these issues, we pro-pose a Hierarchical Global-Local Interaction and Refinement framework for Multimodal Sentiment Analysis (HGLIR). Specifically, a Modality Dropout strategy is first introduced at the input stage to alleviate over-reliance on a sin-gle modality and improve robustness. Based on this, a Hierarchical Local Inter-action (HLI) module models multimodal sequences through a multi-layer pro-gressive structure to capture local dynamic features at different semantic levels. Within the HLI module, a Cross-Modal Synergistic Learning (CMSL) mecha-nism explicitly models cross-modal semantic consistency and gradually aligns information during interaction. Furthermore, a Global Representation Refine-ment (GRR) module introduces learnable global representations and iteratively updates them in a multi-layer structure to aggregate long-range semantic de-pendencies and form stable high-level semantic representations. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate the effectiveness of the proposed framework across multiple evaluation metrics.
yuanyuan zhou· Poster Volume 0007 The 2026...· 0 citations
Multimodal Sentiment Analysis (MSA) aims to predict human sentiment by jointly modeling complementary information from textual, acoustic, and visual modalities. However, effectively exploiting heterogeneous multimodal features remains challenging due to semantic inconsistency, temporal misalignment, and noisy modality-specific representations. To address these issues, this paper proposes a Cross-Attention-based multimodal sentiment analysis framework that explicitly models inter-modal interactions through multi-directional cross-modal attention. Specifically, modality-specific features are first projected into a unified latent space via lightweight modality encoders, after which bidirectional cross-attention is employed to capture complementary dependencies among visual, acoustic, and textual modalities. To further enhance representation learning, multi-head attention and positional encoding mechanisms are incorporated to improve cross-modal interaction modeling and temporal structure awareness. Extensive experiments on the CMU-MOSI benchmark demonstrate that the proposed framework consistently outperforms conventional fusion baselines, including concatenation-based, additive, and self-attention-based fusion strategies. Comprehensive ablation studies further verify the effectiveness of each architectural component, while qualitative attention visualization confirms that the model learns interpretable attention patterns and focuses on semantically informative regions during multimodal fusion. These results indicate that the proposed method provides an effective and interpretable solution for multimodal sentiment analysis and offers useful insights for future attention-based multimodal fusion research.
With the rise of social media, multimodal sentiment analysis has gained increasing attention due to the rich and diverse ways users express emotions through text and images. However, existing methods often ignore fine-grained sentiment cues and semantic misalignment across modalities, limiting this task effectiveness. To address these challenges, we propose DualScope, a novel model that combines a global-local fusion strategy with bidirectional image-text generation for semantically consistent data augmentation. Furthermore, we introduce both label contrastive learning and data contrastive learning to align heterogeneous modalities and enhance model robustness. The fusion module integrates global and local features in a progressive manner under global semantic guidance. Extensive experiments on two widely used datasets, MVSA-Single and MVSA-Multiple, demonstrate that the proposed method achieves superior performance, confirming its effectiveness in handling fine-grained semantics and cross-modal inconsistency.
Bing Zhang, Junteng Wang, Bin Sun et al.· Memetic Computing· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
