Serenify: An AI-Driven Multimodal Digital Coach for Continuous Anxiety and Depression Management

Abstract

Anxiety and depression are among the most prevalent mental health conditions worldwide, yet current clinical assessment tools rely on retrospective self-report, which is subject to recall bias and limited by infrequent clinical contact. Digital health technologies, including mHealth applications and AI-based emotion-detection systems, have emerged to support continuous monitoring, but most rely on manual mood logging or single-modality sensing, capturing only a partial view of the user’s emotional state. To address these limitations, we present Serenify, an AI-driven, multimodal, persuasive mobile application that integrates five passive sensing modalities: facial expression recognition, speech emotion detection, ambient audio classification, text-based emotion inference, and keystroke dynamics analysis. A fusion module aggregates the outputs of dedicated machine learning models into a composite emotional risk level that drives the delivery of personalized wellness interventions. Eight persuasive strategies drawn from the Persuasive Systems Design framework and Goal-Setting Theory are embedded across the application to promote sustained engagement. We evaluated Serenify through an expert evaluation with five Human-Computer Interaction (HCI) experts, using 10 heuristics adapted for persuasive health technologies. The results show an overall mean severity of 0.26 (SD = 0.23), with no rating exceeding cosmetic level, indicating little or no usability problems across the system. Finally, we reflect on our findings to offer three design recommendations for building multimodal persuasive mental health technologies.