Multimodal Emotion Recognition with Emotion-Specific Cross-Modal Attention Blocks

Abstract

Multimodal emotion recognition is increasingly important for healthcare, education, and human-computer interaction. However, many existing systems learn a single shared representation for all emotions, which can blur subtle class-specific cues. This paper proposes an emotion-specific multimodal architecture that combines Wav2Vec speech embeddings, prosody features, and BERT text embeddings through emotion-specific cross-modal attention blocks. Each emotion is assigned a dedicated attention block with its own query projection, while the acoustic modalities are merged using a learnable gated fusion mechanism. In this way, the model learns separate discriminative representations for angry, sad, happy, and neutral emotions rather than relying on a single shared attention path. Experiments on the IEMOCAP corpus show that the proposed model achieves 82.12% unweighted accuracy, outperforming baseline methods. The results indicate that emotion-aware cross-modal attention improves both robustness and class separability in multimodal emotion recognition.