HASF-Net: Hierarchical Attention and Spectral Filtering for Multimodal Emotion Recognition in Conversation

Abstract

Multimodal emotion recognition in conversations (MERC) aims to identify utterance-level emotions from textual, acoustic, and visual cues, and provides a basis for future extension to richer multisensory affective understanding. However, semantic inconsistency across heterogeneous modalities, the under-utilization of modality-specific representations, and the insufficient exploration of latent structural patterns within each modality still limit effective emotion recognition. To address these issues, we propose HASF-Net, a Hierarchical Attention and Spectral Filtering Network for MERC. Specifically, a Hierarchical Modality Encoder is designed to capture intra-modal contextual dependencies and directional inter-modal complementary interactions, producing modality-preserved utterance representations. Then, a Per-modal Fourier Residual Enhancer performs learnable latent spectral filtering to refine modality-specific representations while preserving contextual information. Furthermore, a gated multimodal fusion module adaptively integrates enhanced textual, acoustic, and visual representations for emotion classification. Extensive experiments on the IEMOCAP and MELD datasets demonstrate that HASF-Net achieves competitive performance compared with representative baseline methods. Ablation studies and visualization analysis further verify the effectiveness of hierarchical contextual modeling and per-modal spectral enhancement.