This work presents a system for the Multi-Task Learning (MTL) track of the 11th Affective Behavior Analysis in-the-wild (ABAW) competition on s-Aff-Wild2, the static selected-frame version of Aff-Wild2, showing that post-encoder adaptation and task-wise modeling choices provide a strong MTL pipeline without training a new large-scale face foundation model.
Abstract
Affective behavior recognition in the wild requires joint prediction of continuous valence-arousal, categorical facial expression, and multi-label action units from unconstrained face images. We present our system for the Multi-Task Learning (MTL) track of the 11th Affective Behavior Analysis in-the-wild (ABAW) competition on s-Aff-Wild2, the static selected-frame version of Aff-Wild2. The method focuses on post-encoder adaptation: frozen AffectNet-supervised backbones provide multi-resolution features, while task-specific temporal heads and cross-task fusion modules select the useful signals for each target. For action-unit recognition, we adapt MAE-Face with Low-Rank Adaptation (LoRA) and use DISFA through per-unit expert routing rather than direct sequential transfer. Ablations over backbone, temporal, fusion, and AU-adaptation choices define the final configuration. The final system obtains P = 1.7302 on the official validation split, showing that post-encoder adaptation and task-wise modeling choices provide a strong MTL pipeline without training a new large-scale face foundation model.
Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a $9\times9$ probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points ($T\approx550$, $T\approx2750$). The featured dual-head system reaches 73.0% $\pm$ 0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6% $\pm$ 1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.
Temporal variability limits the practical application of cross-time affective brain-computer interfaces (aBCI). Existing static feature-to-emotion mapping methods are unable to capture the dynamic changes in brain state. Here we propose a multi-level dynamic integrated perception network algorithm (MDIN), which explicitly models temporal variability through a three-layer framework: a temporal-spatial-spectral joint perception module, an adaptive dynamic perception matrix for feature capture, and a discriminative alignment strategy for cross-domain consistency preservation. This structure not only mitigates the impact of temporal variability on emotion recognition but also suppresses noise accumulation in weakly supervised learning scenarios. Experiments on the ECPL and SEED datasets demonstrate that MDIN achieves the highest cross-time recognition accuracies of 96.97% and 94.21%, respectively, outperforming state-of-the-art methods. Notably, as a novel structural solution for temporal variability modeling in affective neuroengineering, it provides a theoretical and technical basis for developing adaptive, long-term stable aBCI systems for mental health monitoring and human-machine interaction.
Feifan Yan, Bo Zhang, Tao Wang et al.· National Science Review· 0 citations
Psychological characteristic estimation from multimodal in-the-wild behavior is usually studied using separate corpora, each annotated for a single target task. Such annotation fragmentation limits cross-task learning and cross-domain generalization across affective, dispositional, and interactional phenomena. To address this problem, we use emotion, apparent personality trait, and ambivalence recognition as representative tasks and introduce MM-PSYCHE, a framework that unifies heterogeneous single-task corpora through cross-domain semi-supervised learning (SSL) with GradNorm-based adaptive task weighting. The framework jointly trains models for emotion, apparent personality trait, and ambivalence recognition. Its multimodal fusion model contains three task-aware components: 1) Task-Specific Projectors, which produce task-conditioned prediction embeddings and re-embed them into a hidden space; 2) Task-Specific Query-Based Cross-Attention Fusion, which uses the prediction embeddings as task-dependent queries and shared modality embeddings as keys and values; and 3) Guide Banks, which are learnable task-specific prototypes that anchor the final predictions. Experiments on the task-specific corpora show competitive supervised learning (SL) performance and more pronounced gains in cross-domain SSL, where heterogeneous corpora can be exploited without joint annotations. On the unseen Multimodal EmotionLines Dataset (MELD), multitask SSL improves generalization in emotion recognition, increasing the macro-F1 score by 7.5 percentage points (35.0% vs. 27.5%) over single-task SSL. The results indicate that cross-domain SSL is useful for fragmented multimodal corpora with heterogeneous psychological annotations, while also showing why single-task SL, single-task SSL, and multitask SSL should be evaluated separately. The source code is publicly available at https://github.com/LEYA-HSE/MM-PSYCHE.
E. Ryumina, A. Axyonov, D. Koryakovskaya et al.· IEEE Access· 0 citations
Constructing Multimodal Emotion Recognition in Conversation (MERC) models is important for understanding affective states from text, audio, images, and video. Existing approaches often rely on linear layers for cross-modal feature alignment and freeze encoder parameters during training, which can introduce feature degradation and modality misalignment. Conventional one-dimensional positional encoding is also limited when modeling visual and temporal information. To address these issues, this study presents Emotion-BIND, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE. The framework maps features from multiple modalities into a shared embedding space and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling. Experimental results show that Emotion-BIND achieves an Unweighted Average Recall (UAR) of 49.44%, a Weighted Average Recall (WAR) of 71.00%, and an F1 score of 90.05% on the DFEW dataset, with the highest WAR and F1 score among the compared methods, although its UAR remains lower than the best comparison model. On the MELD dataset, Emotion-BIND achieves a UAR of 55.67%, a WAR of 67.88%, and an F1 score of 91.02%. The source code is archived at https://doi.org/10.5281/zenodo.21186484 and is also available on GitHub at https://github.com/qunlitang/Emotion-BIND.
Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition. EEG provides millisecond-level access to neural activity, yet most EEG pipelines analyze the signal through a single temporal window, thereby fixing the temporal structure available to the model. This study introduces a multi-scale temporal framework for EEG-based emotion recognition. The EEG waveform is decomposed into windows of one or several durations, processed by a shared attention-based encoder, and integrated through a dynamic fusion module that assigns sample-specific weights across temporal scales. The framework is evaluated under a subject-independent protocol in binary and three-class settings, with the three-class task including the mixed affective category. The best results are 65.22% for the two-class task and 45.43% for the three-class task. Both are obtained with three-scale dynamic-fusion configurations and remain substantially above the full-signal baseline. The best-performing temporal scales differ between the two tasks. Dynamic fusion outperforms concatenation in the highest-scoring two-class configuration and slightly exceeds it in the highest-scoring three-class configuration, although these multi-scale settings require substantially more computation than the full-signal baseline.
Stefanos Gkikas, Yanglu Guo, Guangliang Li et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
