Skip to content
Open access

A Reliability Sensitive Theory of Multimodal Emotion Recognition

Oct 2026 · Applied and Computational Engineering · 0 citations

TL;DR

A reliability-sensitive framework has been proposed, in which recognition is driven by uncertainty management rather than feature accumulation, while preserving the modality-specific nature of different types of evidence.

Abstract

Current research on multimodal emotion recognition seeks to analyse the relationships among facial, vocal, and bodily expressions in order to recognise different emotions. Although additional data streams are often introduced, they cannot be treated as independent evidence. This conceptual study aims to understand the reliability of multimodal inferences over unimodal classifications. This study covers the theory and practice of affective computing, the study of different modalities, fusion taxonomies and benchmarking. Depending on the statistical dependencies that they learn, fusion can occur in early, intermediate, late or adaptive form. The synthesis suggests that the advantage of multimodal systems is determined by four factors: informational complementarity, partially independent errors, temporally valid alignment, and dynamic reliability estimation. For these factors, fusion of different types of data is a good alternative. Otherwise, it is likely to increase the recording of social displays, bias of annotations, or dependency on a dominant modality. For this reason, a reliability-sensitive framework has been proposed, in which recognition is driven by uncertainty management rather than feature accumulation, while preserving the modality-specific nature of different types of evidence. The resulting account explains both the recurrent gains reported in benchmark studies and their contraction under naturalistic expression, distribution shift, cultural mismatch, strategic display, and systematic missingness. Multimodal systems become defensible not by sensing more behaviour alone, but by estimating when each observation should change, confirm, or weaken an affective inference.

Read PDF

Similar papers

Open access Sep 2026

Multimodal Emotion Recognition System Using Facial Expressions and Speech Analysis

Human communication involves emotion as an important issue. Because of emotion, people express and understand feelings, attitudes, and actions. However, effective automatic emotion recognition is still difficult to achieve when based on one single cue, as for instance facial expressions or voice may both present weak o...

S. G., N. S · 0 citations
Review Open access Sep 2026

Towards Unified Affective AI: A Cross-Modal Survey of Emotion Recognition, Generation, and Control

This survey provides a comprehensive review of state-of-the-art methodologies for emotion recognition and generation across facial, speech, and textual modalities, covering preprocessing techniques, datasets, deep learning architectures, evaluation metrics, and emotion control mechanisms.

Rebecca Mobbs, Dimitrios Makris, V. Argyriou · 0 citations
Sep 2026

CLMER: A Framework for Contrastive Learning-Based Multimodal Emotion Recognition.

Experimental evaluations on two public datasets DEAP, AMIGOS and a private dataset MAN-II demonstrate that CLMER significantly outperforms unimodal and traditional fusion approaches, achieving state-of-the-art performance in emotion classification tasks.

Shuang Niu, Jian He, Yu Liang et al. · 0 citations
Book Open access Oct 2026

Recognition of Affective States Using Multimodal and Adaptive AI for Individuals with Motor Disabilities

People with motor disabilities often display atypical emotional expressions that elude conventional recognition models, leading to communication difficulties, isolation, and restricted access to emotional support. Multimodal AI affective recognition systems offer promising avenues to overcome these barriers, but existi...

Maram Djebbi, Kirmene Marzouki, Fernando Marmolejo-Ramos et al. · 0 citations
Review Open access Oct 2026

A Unified Perspective on Multimodal Emotion Recognition, Personality Trait Assessment, and Ambivalence/Hesitancy Recognition: Corpora, Methods, Evaluation, and Open Challenges

Findings indicate that many methodological trends remain supported primarily by within-benchmark evidence, and the synthesis identifies shared multimodal processing principles while preserving differences in target stability, annotation sources, temporal granularity, and evaluation.

Angelina Egorova, Timur Abdulkadirov, E. Ryabova et al. · 0 citations
Review Open access Oct 2026

Faces and bodies in emotion recognition: complementary and conflicting cues

The face has traditionally been considered the primary source of information for emotion recognition. However, in everyday social interaction, facial expressions are embedded within bodies, actions, scenes, and interactions. This review examines how facial and bodily expressions shape emotion recognition, focusing on w...

Yuan-Yi Peng, A. Kafkas, Karen Lander · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.