Designing Multimodal Conversational AI for Longitudinal Well-Being Monitoring

Abstract

Traditional mental health assessment often relies on infrequent self-report questionnaires (e.g., PHQ-9), which provide sparse, retrospective snapshots of well-being and may miss critical daily fluctuations. This creates an opportunity for low-burden assessments embedded in everyday interactions, such as open-ended conversations with AI systems, to complement existing clinical approaches. This doctoral research investigates how conversational AI agents powered by Large Language Models (LLMs) can predict well-being states by analyzing multimodal behavioral cues. It follows a four-stage plan: establishing a completed LLM-based baseline for predicting well-being from open-ended video responses; analyzing how LLMs integrate or misinterpret cross-modal signals; designing conversational flows that support response quality and user comfort; and evaluating longitudinal interactions for capturing intra-individual well-being fluctuations. Together, this work contributes a human-centered framework for multimodal well-being monitoring through natural conversations that complements, rather than replaces, traditional mental health assessment.