CMF-LLaVA: Causal Modality-Aware Fusion with Effect Estimation for Multimodal Aspect-Based Sentiment Analysis
Abstract
Multimodal Aspect-Based Sentiment Analysis (MABSA) predicts sentiment toward a specified aspect from paired text and images. In realistic multimodal posts, however, the two modalities are often asymmetric in reliability and may even provide conflicting evidence for the queried aspect. Correlation-based fusion is therefore vulnerable to dataset-specific shortcuts and unreliable modality cues. To address this problem, we propose CMF-LLaVA, a causally inspired modality-aware extension of LLaVA for MABSA. The core of the model is a Causal Disentanglement and Attribution (CDA) module that performs four steps: (1) reduces readily exploitable label cues through adversarial regularization, (2) derives modality-effect surrogates by contrasting aspect-conditioned representations with learnable null references, (3) reduces aspect--context dependence using HSIC, and (4) performs reliability-aware FiLM fusion. Semantic baseline regularization constrains the null references to induce high-uncertainty predictions, preventing arbitrary latent offsets. The resulting effects are interpreted as intervention-inspired representation contrasts rather than formally identified causal quantities. We further introduce the Modality Attribution Dataset (MAD), a human-annotated benchmark with substantial inter-annotator agreement (Krippendorff's ?=0.81), to evaluate agreement between model attributions and human judgments. On Twitter-15, Twitter-17, and MASAD, CMF-LLaVA outperforms strong MABSA and MLLM baselines and exhibits smaller performance drops under modality conflict, spurious visual cues, and aspect ambiguity. These results support causally inspired effect modeling for robust and interpretable MABSA.