Embracing Unimodal Aleatoric Uncertainty for Multimodal Emotion Recognition in Conversations

Abstract

In conversational emotion recognition, every utterance is assigned an affective label using the textual, acoustic, and visual streams that accompany speech; however, each single-modality representation is typically contaminated by noise and ambiguous semantics, which harms cross-modal interaction and weakens fusion. We investigate an uncertainty-aware fusion architecture for the 7-way emotion classification task on the MELD benchmark. Our network represents every modality by a Gaussian whose variance measures its uncertainty, exchanges information along both the modality and feature dimensions through a CubeMLP block, models cross-modal relations with a multi-head attention layer, and combines the streams via uncertainty-driven dynamic weights before the classifier. A class-imbalance-aware loss keeps training stable under MELD’s heavily skewed label distribution. The model reaches an accuracy of 62.65±0.24% and a weighted-F1 of 65.29±0.13%.