Multimodal Deep Learning for Depression Detection using Smartphone Features
Abstract
Depression is a common mental health disorder affecting emotional, behavioral, social, and cognitive well-being. Traditional assessment relies on clinical interviews and self-reported questionnaires, which are subjective and fail to capture daily behavioral changes. To overcome this, we propose a multimodal deep learning system for automatic depression detection using mobile sensing and task-based behavioral data. The framework has three stages: data collection, multimodal feature extraction, and model training. Data are collected via participatory mobile sensing, task-based experiments, and ground-truth assessment. Preprocessed data yield low-level features like smartphone usage and audio-visual characteristics, followed by high-level statistical features. Relevant features are fused to represent behavioral and psychological patterns. Deep learning models, including CNNs and RNNs, are trained on fused features to classify individuals as depressed or non-depressed. The system offers an objective, scalable approach for early screening and continuous monitoring. Integrating multiple behavioral modalities with deep learning enhances detection robustness over single-source methods.