Emotion-Aware Virtual Reality Through Multimodal ECG and Postural Fusion

Abstract

Immersive Virtual Reality (VR) environments are increasingly adopted in clinical psychology and stress-management contexts; however, their therapeutic effectiveness depends on the system’s ability to understand and dynamically respond to users’ affective states. Emotional regulation plays a key role in psychological resilience, directly influencing stress coping mechanisms, cognitive performance, and overall mental well-being. Despite recent advances, automatic recognition of scenario-associated affective conditions in VR remains challenging because head-mounted displays occlude facial features. This study proposes a VR-based serious game for affective training and regulation, in which users interact with goal-oriented scenarios targeting fear, anger, and joy. We introduce a multimodal affective computing model to objectively assess users’ emotional responses by integrating electrocardiogram (ECG) signals and posture-based features extracted through computer vision. An early-fusion architecture combined with a Long Short-Term Memory (LSTM) network captures temporal dependencies in synchronized multimodal data. We established a controlled experimental framework using immersive VR scenarios, enabling the collection of synchronized physiological and behavioral data from a cohort of 20 healthy adult participants. The proposed model was evaluated under a strict Leave-One-Subject-Out (LOSO) cross-validation scheme across independent subjects, achieving a robust inter-subject accuracy of 78.94%±10.77% and a global macro F1-score of 0.635, demonstrating strong generalization to entirely unseen users without data leakage. Furthermore, the system maintained an outstanding balance in detecting active emotional states (recall > 80.0% for fear, anger, and joy). Additionally, subjective evaluations using the PANAS and SGU questionnaires confirmed the coherence between detected and perceived emotional states, as well as the system’s high usability. The results suggest the potential viability of combining immersive environments and multimodal affective computing to explore the technical feasibility of adaptive frameworks that could eventually translate into healthcare contexts. This work may contribute to the development of intelligent digital health technologies by providing a foundation for responsive VR systems that can monitor emotional regulation and are fully aligned with sustainable well-being ecosystems (SDG 3: Good Health and Well-being and SDG 10: Reduced Inequalities).