Generative AI emotion recognition from bodily gestures and vocal tone reveals modality-specific performance and positivity bias

Abstract

Advances in generative artificial intelligence (GenAI) have prompted interest in its application in sensitive fields, including mental health. Yet the validity of its nonverbal social-cognitive abilities remains unclear. This study addresses two issues that probe GenAI’s social intelligence: its capacity to interpret nonverbal channels (bodily gestures and vocal tone) with accuracy comparable to humans, and whether accuracy of emotion recognition is different by emotional valence. Emotion recognition accuracy of three Gemini GenAI models (Pro 1.5, Pro 2 (Experimental) and Flash 2) were evaluated using the EU-Emotion Stimulus Set of bodily gestures and vocal tone depicting basic emotions. Model accuracy was compared to validated human benchmarks. Advanced GenAI models achieved near-human accuracy in recognizing emotions from vocal tone. However, their performance was significantly lower than that of humans in interpreting bodily gestures. GenAI models recognized positive emotions more accurately than negative ones. This bias was significant across all models for bodily gestures and in the advanced models for vocal tone. GenAI models exhibit a non-humanlike social-cognitive profile, excelling with positive emotions but struggling with the negative content. This finding carries profound clinical and theoretical relevance, highlighting the risks of integrating GenAI into sensitive fields and underscoring the need for continued validation.