An IoT-enabled multimodal AI system for classifying adolescent mental states using facial, motion, and postural data
Abstract
This paper presents the design, development, and evaluation of a human-centric smart mental environment that integrates internet of things (IoT)-enabled sensors and artificial intelligence (AI) for the early detection of adolescent mental health indicators. Adolescent mental health disorders, including stress, anxiety, and depression, are increasingly prevalent yet remain underdiagnosed due to reliance on subjective self-reports and clinical interviews. Early detection is critical but hindered by symptom underreporting and the absence of continuous, objective monitoring tools suitable for real-world environments. To address this gap, the proposed system captures three complementary modalities: (1) facial expression features using a front-facing camera, (2) body motion and acceleration patterns using a wearable accelerometer and gyroscope, and (3) postural pressure distribution using an eight-element pressure array positioned along the thoracic and lumbar regions. Multimodal data are preprocessed using temporal-spatial filtering, followed by a cross-modal attention-based fusion mechanism that dynamically weights the most informative modality within each time window. A supervised learning classifier then predicts four mental states: normal, stress, anxiety, and depression. The system was evaluated using a dataset collected from 64 adolescents aged 14–18 under semi-natural, classroom-like conditions. The proposed attention-based multimodal framework achieved an overall classification accuracy of 89.2%, significantly outperforming unimodal baselines (facial-only: 77.8%, motion-only: 74.5%, posture-only: 71.3%) and simple concatenation fusion (83.5%). The attention mechanism improved the classification F1-score to 0.88 and demonstrated robustness under lighting and posture variations. Statistical analysis confirmed the superiority of the multimodal approach (p < 0.01). The system provides a non-invasive, scalable, and human-centric solution for continuous adolescent mental health monitoring, with strong potential for deployment in schools and community health programs.