Adaptive Multimodal AI Tutoring: Using Real-Time Cues to Personalize Feedback and Support Engagement in Learning

Abstract

Learner engagement is commonly viewed as a key factor in successful learning. In online settings, limited face-to-face interaction can make learners more prone to distraction and reduced attention, highlighting the importance of monitoring and sustaining engagement. Recent advances in generative AI allow systems to infer learners’ cognitive and emotional states from multimodal cues, enabling more personalized and adaptive instructional support. However, little research has examined how such systems can dynamically adapt both the type and timing of feedback based on learners’ moment-to-moment engagement states inferred from multimodal signals. This work presents an adaptive multimodal AI-driven tutoring system that infers learners’ states by interpreting real-time visual, auditory, and behavioral cues. Based on the inferred learner state, the AI tutor determines when and how to intervene to sustain engagement. The system is structured as a closed-loop cognitive architecture: perception (capturing real-time multimodal cues), decision (aggregating the multimodal inputs into four affective metrics), and action (delivering feedback based on the inferred state by mapping each metric to a feedback type and timing strategy). This work presents a high-fidelity, interactive multimodal AI tutoring system that illustrates the feasibility of integrating multimodal cues to enable adaptive instructional feedback and engagement-aware intervention in online learning contexts.