Anxiety and depression are among the most prevalent mental health conditions worldwide, yet current clinical assessment tools rely on retrospective self-report, which is subject to recall bias and limited by infrequent clinical contact. Digital health technologies, including mHealth applications and AI-based emotion-detection systems, have emerged to support continuous monitoring, but most rely on manual mood logging or single-modality sensing, capturing only a partial view of the user’s emotional state. To address these limitations, we present Serenify, an AI-driven, multimodal, persuasive mobile application that integrates five passive sensing modalities: facial expression recognition, speech emotion detection, ambient audio classification, text-based emotion inference, and keystroke dynamics analysis. A fusion module aggregates the outputs of dedicated machine learning models into a composite emotional risk level that drives the delivery of personalized wellness interventions. Eight persuasive strategies drawn from the Persuasive Systems Design framework and Goal-Setting Theory are embedded across the application to promote sustained engagement. We evaluated Serenify through an expert evaluation with five Human-Computer Interaction (HCI) experts, using 10 heuristics adapted for persuasive health technologies. The results show an overall mean severity of 0.26 (SD = 0.23), with no rating exceeding cosmetic level, indicating little or no usability problems across the system. Finally, we reflect on our findings to offer three design recommendations for building multimodal persuasive mental health technologies.
Harsh Maisuri, Josteve Adekanbi, R. Orji et al.· International Conference on...· 0 citations
Emotional well-being is fundamental to mental health, yet many individuals find it difficult to recognize and manage their emotional states in daily life. In 2022, the World Health Organization (WHO) reported that more than 1 billion people are currently living with emotional disorders, with conditions such as anxiety and depression placing a significant burden on both individuals and society. Current digital emotional wellness tools rely on manual self-report mechanisms and offer predefined emotion options for users to choose from, which limit their capacity to support personalized interventions and influence mood change. To address these limitations, this study presents ReflectAI, a mobile application that uses multimodal emotion detection to accurately capture the emotional state of the user and deliver personalized interventions for emotional well-being. ReflectAI captures the emotional state of the user through three input modalities: typed text, voice recordings, and facial images. Each input is then processed by a dedicated artificial intelligence model to predict the emotional state of the user along with a confidence score. Based on the emotions detected, the system delivers four personalized interventions, namely motivational quotes, music recommendations, meditation videos, and breathing exercises, to support mood regulation. To evaluate ReflectAI, an expert evaluation was conducted with 5 Human-Computer Interaction (HCI) experts using 10 heuristics adapted for persuasive health technologies. The results show an overall mean severity of 0.32 (SD = 0.47), with no rating exceeding the cosmetic level, indicating little or no usability problems across the system. The findings are used to inform a set of practical design guidelines for researchers who seek to develop future emotional awareness applications that integrate multimodal emotion detection with personalized intervention delivery to promote emotional well-being.
Judith Kurian, Anjali Rachel Benjamin, Gladwin Irudayaraj et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
