Uncertainty Aware Multimodal Framework for Fine-Grained Human Emotion Grading

Abstract

Human emotion analysis is central to applications such as healthcare monitoring, human–computer interaction, education, and intelligent decision support systems. Existing emotion recognition approaches largely classify emotions into discrete categories rather than estimating fine-grained intensity levels, and typically rely on a single modality, reducing robustness under real-world conditions. Furthermore, most existing models lack uncertainty awareness, producing predictions without confidence estimates, which limits their reliability when inputs are noisy, ambiguous, or incomplete. This paper proposes an uncertainty-aware multimodal emotion grading framework that integrates facial, speech, physiological, and behavioral cues through an attention-based fusion mechanism to jointly estimate emotion category, fine-grained intensity level, and prediction confidence. The proposed architecture combines modality-specific deep encoders, a cross-modal fusion module, an ordinal emotion-intensity grading head, and an uncertainty estimation module based on Monte Carlo Dropout and evidential learning principles. A proof-of-concept implementation on standard public affective computing datasets (RAVDESS, IEMOCAP, DEAP) achieves a classification accuracy of 87.4%, an F1-score of 0.86, an intensity-grading MAE of 0.091, and an RMSE of 0.127, with predicted uncertainty exhibiting a correlation of 0.81 with actual prediction error, confirming that the estimated confidence scores are meaningful. These preliminary results establish the architectural foundation for subsequent large-scale performance evaluation and against unimodal and non-uncertainty-aware baselines.