Modeling Affective Distress from Video: A Modality-Aware Multi-Branch Learning Approach

Abstract

Automated detection of mental health conditions such as stress and Social Anxiety Disorder (SAD) from video is an important yet challenging problem, as these states exhibit both transient micro-expressions and sustained behavioral patterns over time. While stress and anxiety are distinct constructs, they share overlapping behavioral manifestations; in this work, we study them under a unified framework of affective distress. Existing video-based approaches often process heterogeneous behavioral cues using a single sequential encoder, which can lead to information loss when modeling signals with diverse temporal characteristics. To address this limitation, we propose a Multi-Branch Modality-Aware Multiple Instance Learning (MIL) framework that explicitly disentangles behavioral features based on their temporal dynamics. Specifically, high-variance and sparse signals (e.g., facial Action Units) are processed using a convolutional branch to capture transient behavioral patterns, while temporally continuous signals (e.g., postural landmarks and gaze) are modeled using a Bidirectional LSTM to capture long-term trends. A gated attention mechanism aggregates window-level representations into participant-level predictions while providing interpretable temporal saliency maps. We evaluate our approach across four datasets capturing stress- and anxiety-related conditions, including a private SAD dataset, StressID, UBFC, and AVEC 2017. Results demonstrate competitive or improved performance over state-of-the-art methods, along with strong cross-dataset generalization. Additionally, we introduce a gradient-based video rendering pipeline that overlays model attention onto input videos, enabling interpretable spatial and temporal insights for assessment. Our implementation is publicly available at https://github.com/OmGovind15/MAMBL.