Research on Emotion-Aware Multimodal Input Adaptation Strategies in Augmented Reality Environments

Abstract

This study investigates how emotion-aware multimodal input adaptation can improve interaction stability in augmented reality environments. In AR tasks, users rely on gesture, gaze, speech, and touch-based input, but these modalities become less reliable under stress, confusion, or fatigue. To address this issue, an emotion-aware adaptation framework is developed, integrating multimodal sensing, emotion recognition, and real-time input adjustment. The prototype is implemented using Microsoft HoloLens 2, Unity 2022.3 LTS, MRTK 2.8, Python 3.10, PyTorch 2.1, Flask, and PostgreSQL 15. User data include gaze behaviour, hand gestures, speech features, heart rate, electrodermal activity, skin temperature, and interaction logs. A late-fusion BiLSTM model classifies users into neutral, stress, confusion, and fatigue states. Based on recognition results, the system dynamically adjusts input priority, confirmation rules, visual cues, menu complexity, and task guidance. Experimental results show that recognition accuracy reaches 88.42%, task completion time is reduced by about 14%, input error rate decreases by about 39%, and perceived workload is significantly lowered under adaptive input conditions.