Progressive Evolution of Emotion Detection: From Unimodal Baselines to a Quad-Modal Dynamic Fusion Architecture

Abstract

The existing emotion detection systems are either audio or video centric. While such approaches have proven effective in controlled environments like a lab or a studio, these systems fail to perform under real-world conditions such as low light, audio interference, mispronunciation, and other environmental factors. The objective of our research is to design a quad-modal emotion detection system that combines four sub-models (static facial images, dynamic facial expressions, speech tone, and natural language processing [NLP] of the spoken words) to design a robust model that can perform under real-world conditions. We propose a dynamic weighting algorithm that biases the model towards a certain modality if the conditions are favorable (e.g. switch to audio-only if the video is obstructed) using cross-modal multi-head attention. We optimise the system to run on a CPU (due to the lack of a CUDA-enabled NVIDIA GPU) by using an asynchronous state buffer for the heavy NLP pipeline to maintain a high enough video frame rate. Our model achieved an accuracy of 61.21% (95% confidence interval [CI]: 58.77%-63.65%) with a mean area under the curve of 0.90 (95% CI: 0.88-0.92) across six emotion classes on real-world data (RAVDESS, SAMM, CREMA-D) during training and testing while maintaining performance under visual and audio deprivation conditions. Our architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.