Bidirectional joint cross-attention framework for transformer based audio–visual emotion recognition

Lightweight and robust audio-visual emotion recognition via multi-scale mamba temporal modeling and quality-aware expert fusion

Multimodal emotion recognition for real-time human–computer interaction requires both high recognition accuracy and low computational cost. However, existing audio-visual methods often rely on heavy Transformer-based fusion or high-capacity visual backbones, making them difficult to deploy on edge devices. Moreover, their robustness to degraded audio-visual inputs and their class-wise behavior for ambiguous emotions remain insufficiently analyzed. To address these issues, we propose a lightweight audio-visual emotion recognition framework. The visual stream uses ShuffleNet for efficient facial feature extraction, while the audio stream uses multiscale MFCCs and efficient sequence modeling to capture emotional prosody. A cross-modal auxiliary fusion module is further introduced to align audio and visual representations, and lightweight channel attention is used to emphasize emotion-relevant features. Experiments on CREMA-D and IEMO-CAP demonstrate that the proposed method achieves competitive recognition accuracy with significantly fewer parameters and lower computational cost. Additional analyses on edge-device inference, audio-visual degradation, class-wise confusion, and batch-size sensitivity further validate the efficiency and robustness of the proposed framework.

MULTI-MODAL TRANSFORMER ARCHITECTURE WITH CROSS-ATTENTION FUSION FOR ROBUST AUDIO-VISUAL SENTIMENT ANALYSIS

The proposed architecture provides an effective and interpretable framework for robust multimodal sentiment analysis and offers a promising foundation for real-world affective computing applications.

Multimodal Emotion Recognition in Urdu through Late Fusion of Fine-Tuned Speech and Text Representations

This study proposes a multimodal deep learning framework for Urdu emotion recognition by integrating speech and text modalities that surpasses the existing UMEDNet benchmark, demonstrating the effectiveness of transformer-based feature extraction and multimodal late fusion for Urdu emotion recognition.

Advancing emotion recognition: a novel adaptive audio–visual fusion approach

Cross-Modal Attention Mechanisms in Transformer Networks for Real-Time Emotion Detection in Video Conferencing

Emotion detection in video conferencing requires models that can interpret facial-expression dynamics and speech cues under variable communication conditions. This study develops a reliability-aware RA-Gated Cross-Modal Fusion framework for real-time audio-visual emotion recognition using the RAVDESS full audio-visual speech subset. The methodology includes synchronized audio-visual preprocessing, face-frame extraction, log-mel spectrogram generation, unimodal baseline training, cross-modal feature extraction, and reliability-aware gated fusion. Two evaluation protocols were used: a stratified-random split for main benchmark assessment and an actor-independent split for speaker-disjoint generalization. The RA-Gated Cross-Modal Fusion model achieved the strongest stratified-random performance, reaching 93.55% accuracy, 93.14% macro-F1, and 93.59% weighted-F1, outperforming audio-only, visual-only, and late-fusion baselines. Class-wise analysis further showed strong recognition of angry, fearful, disgust, happy, and improved performance for sad and surprised expressions. The actor-independent evaluation showed lower performance across all models, with tuned late fusion achieving the highest speaker-disjoint result and RA-Gated fusion improving over unimodal baselines. These findings indicate that reliability-aware fusion improves benchmark emotion recognition, while the 21.50 ms per-sample RA-Gated latency on an NVIDIA GeForce RTX 3060 supports real-time-oriented applicability. Future work should test larger naturalistic conferencing datasets and strengthen speaker-invariant streaming transformers.

Latency-Aware Hybrid Transformer–Capsule Network for Audio-Visual Emotion Recognition in Edge–Fog–Cloud Environments

Audio-visual emotion recognition (AVER) is central to affective computing systems that require reliable, real-time interpretation of human emotions. However, many existing multimodal models treat feature learning and deployment efficiency separately, limiting their ability to preserve hierarchical facial relationships, capture long-range speech dynamics, and operate with low latency in distributed settings. This study proposes a latency-aware hybrid Transformer–capsule network for audio-visual emotion recognition in a simulated edge–fog–cloud environment. The visual stream employs a CNN–Capsule branch to retain spatial hierarchies in facial expressions, while the audio stream uses a CNN–Transformer branch to learn local spectral patterns and long-range temporal dependencies from speech. A cross-modal Transformer fusion module integrates complementary emotional cues, and a latency-aware task-allocation mechanism allocates preprocessing, inference, and training-related operations across edge, fog, and cloud layers according to workload, node capacity, and communication delay. Unlike approaches that optimize multimodal representation learning and distributed deployment as separate problems, the proposed framework adopts a deployment-aware co-design in which spatial visual representation, temporal acoustic modeling, multimodal interaction, and deterministic latency-aware task allocation are coordinated within a unified processing pipeline. The framework is evaluated on RAVDESS, CREMA-D, and SAVEE using a subject-independent protocol. Experimental results show an average accuracy of 91.5%, an F1-score of 90.7%, an MCC of 0.894, and an AUC of 0.950. The framework further incorporates a deterministic latency-aware task-allocation mechanism for coordinating operations across edge, fog, and cloud resources. Physical-device deployment and comprehensive resource profiling remain subjects for future validation.