Skip to content
Review Open access

A Unified Perspective on Multimodal Emotion Recognition, Personality Trait Assessment, and Ambivalence/Hesitancy Recognition: Corpora, Methods, Evaluation, and Open Challenges

Oct 2026 · Big Data and Cognitive Computing · 0 citations · 126 references

TL;DR

Findings indicate that many methodological trends remain supported primarily by within-benchmark evidence, and the synthesis identifies shared multimodal processing principles while preserving differences in target stability, annotation sources, temporal granularity, and evaluation.

Abstract

Multimodal analysis of human behavior has advanced rapidly, yet emotion recognition (ER), personality trait assessment (PTA), and ambivalence/hesitancy recognition (AHR) are usually reviewed separately. This review jointly examines these related but distinct tasks through a comparative analysis of corpora, computational methods, evaluation protocols, and ethical limitations. Structured searches in Scopus and Web of Science were followed by multi-stage screening and multi-reviewer assessment. The final methodological sample comprised 56 unique studies published between 1 January 2023 and 1 August 2026. One study addressed both ER and PTA, resulting in 57 task-specific entries. The reviewed methods include cross-modal interaction, temporal and contextual modeling, graph-based reasoning, reliability-aware fusion, learning under limited supervision, and semantic enrichment with large language models. The reviewed ER studies address contextual modeling, robustness-related objectives, and open-vocabulary prediction. PTA studies focus on structured behavioral modeling, temporal aggregation, target-aware assessment, and generalization-related strategies. AHR remains less explored and is concentrated on a narrow benchmark landscape with distinct frame- and video-level protocols. A post-inclusion quality and reproducibility assessment showed explicit repeated-run uncertainty reporting in only 10 of 56 studies. Explicit cross-corpus or external validation was reported in only 3 of 56 studies. These findings indicate that many methodological trends remain supported primarily by within-benchmark evidence. The synthesis identifies shared multimodal processing principles while preserving differences in target stability, annotation sources, temporal granularity, and evaluation. Key priorities include cross-corpus generalization, uncertainty, interpretability, cultural validity, privacy, and human oversight.

Read PDF

Similar papers

Open access Oct 2026

A Reliability Sensitive Theory of Multimodal Emotion Recognition

A reliability-sensitive framework has been proposed, in which recognition is driven by uncertainty management rather than feature accumulation, while preserving the modality-specific nature of different types of evidence.

Yi-Hang Chen · 0 citations
Conference Open access Sep 2026

GPT-5’s Reasoning Process in Emotion Recognition: A Content Analysis Approach

While Large Language Models (LLMs) demonstrate strong performance in textual emotion detection, their underlying reasoning mechanisms remain unclear. This study investigates the reasoning processes of GPT-5 by conducting a content analysis of its generated justifications for Weibo posts. Using the GPT API, we generated...

Fa Zhang, Qiu-Yue Wu · 0 citations
Review Open access Sep 2026

Beyond Associations Between Color and Emotion: A Scoping Review of Task-Based Evidence on Color Properties, Emotional Attention, and Cognitive Control

Color may influence emotional cognition through perceptual salience, learned affective meaning, and response compatibility, yet task-based evidence has not been synthesized. Reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for Scoping Reviews (PRISMA-ScR) and the...

Chao-Sheng Tang, Tian-Ning Gu, Jun-Yi Hua et al. · 0 citations
#large language models Open access Oct 2026

A multi-model LLM framework for AI-assisted psychological scale development: initial psychometric evaluation of the Emotion Regulation Scale (ERS)

Emotion regulation is a key construct in mental health, yet existing measures differ in conceptual coverage, with some focusing on specific strategies or regulatory difficulties rather than broader cognitive, behavioral, and contextual regulation processes. This study aimed to develop an Emotion Regulation Scale (ERS)...

I. Almohammadi, M. Alzahrani · 0 citations
Open access Oct 2026

The emotion regulation and recognition account of empathy-related processes in autism: a multimodal, multilevel case-control study.

BACKGROUND Recent evidence suggests that many autistic individuals exhibit heightened emotional responsiveness to others' affect despite greater difficulty recognizing others' emotions and accurately inferring their perspectives. This study introduces and preliminarily evaluates the Emotion Regulation and Recognition A...

Jeffrey J. Wood, Stian Orm, Karen S. Wood et al. · 0 citations
Open access Oct 2026

Development of the Facial Emotion Recognition Task (TAREFE): methodological aspects and initial psychometric indicators

Facial Emotion Recognition (FER) is a core component of social cognition, supporting theory of mind processes, empathy, and social reasoning. Deficits in FER are associated with functional impairments across multiple domains of typical and atypical development. Measurement instruments with sound psychometric prop...

F. de Lima Osório, Diego Alves Ferreira, Nelson Torro-Alves · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.