Multimodal Affective AI for Engagement and Stress-Aware Adaptive Human Robot Interaction

Abstract

This paper presents a multimodal affective AI framework for engagement- and stress-aware adaptive human–robot interaction (HRI) using a simulated Pepper robot. The proposed system integrates real-time facial expression recognition, webcam-based remote photoplethysmography, neuro-affective language understanding, Russell circumplex affective modeling, and Bayesian adaptive robot feedback learning within a unified closed-loop architecture. Two affective AI modules are introduced: HySpaN, a hybrid hyperbolic transformer-based facial expression recognition framework, and a neuro-affective BERT model for sentiment, emotion, psychological, behavioral, and risk-aware language interpretation. A proof-of-concept experimental evaluation involving seven participants demonstrated the capability of the framework to continuously estimate multimodal affective states, regulate adaptive robot behaviors, and analyze engagement, stress, frustration, and emotional trajectories during interaction. The results showed meaningful affective variations across experimental blocks, competitive performance of the proposed affective perception models on public datasets, and promising adaptive regulation trends through Bayesian Immediate Feedback Learning. Although evaluated using a simulated Pepper robot and a proof-of-concept participant cohort, the proposed framework establishes a robust foundation for future large-scale validation with physical humanoid robots and more statistically representative populations, demonstrating the potential of multimodal affective AI for adaptive, personalized, and emotionally-aware HRI systems.