Skip to content
Book Open access

CMF-LLaVA: Causal Modality-Aware Fusion with Effect Estimation for Multimodal Aspect-Based Sentiment Analysis

Nov 2026 · 0 citations

Abstract

Multimodal Aspect-Based Sentiment Analysis (MABSA) predicts sentiment toward a specified aspect from paired text and images. In realistic multimodal posts, however, the two modalities are often asymmetric in reliability and may even provide conflicting evidence for the queried aspect. Correlation-based fusion is therefore vulnerable to dataset-specific shortcuts and unreliable modality cues. To address this problem, we propose CMF-LLaVA, a causally inspired modality-aware extension of LLaVA for MABSA. The core of the model is a Causal Disentanglement and Attribution (CDA) module that performs four steps: (1) reduces readily exploitable label cues through adversarial regularization, (2) derives modality-effect surrogates by contrasting aspect-conditioned representations with learnable null references, (3) reduces aspect--context dependence using HSIC, and (4) performs reliability-aware FiLM fusion. Semantic baseline regularization constrains the null references to induce high-uncertainty predictions, preventing arbitrary latent offsets. The resulting effects are interpreted as intervention-inspired representation contrasts rather than formally identified causal quantities. We further introduce the Modality Attribution Dataset (MAD), a human-annotated benchmark with substantial inter-annotator agreement (Krippendorff's ?=0.81), to evaluate agreement between model attributions and human judgments. On Twitter-15, Twitter-17, and MASAD, CMF-LLaVA outperforms strong MABSA and MLLM baselines and exhibits smaller performance drops under modality conflict, spurious visual cues, and aspect ambiguity. These results support causally inspired effect modeling for robust and interpretable MABSA.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.