Interpretable Multimodal Engagement Prediction with Graph-based Mixture-of-Experts

Abstract

Automatic engagement prediction is a significant aspect of Human-Computer Interaction (HCI) and affective computing, enabling systems to capture user interest and deliver timely interventions. In dyadic conversations, a person’s (target’s) engagement can be influenced by both their own behavioral signals and those of their interaction partner. This motivates the need for models that not only capture these influences but also explain the self and social signals’ contributions towards engagement prediction. To this end, we propose a graph-based Mixture-of-Experts (MoE) framework that explicitly separates self and social influences to enable interpretable engagement modeling. Our framework comprises two Graph Neural Network (GNN) based experts: (i) a self-branch that models engagement from the target’s own multimodal signals, and (ii) a social-branch that captures interaction dynamics by integrating signals from both participants. Within the social branch, nodes represent participants with unsupervised social roles learned from multimodal representations, while role-conditioned edge weights highlight influential interactions. A gating mechanism adaptively combines expert predictions, providing insight into the relative contributions of self and social factors. To further enhance robustness and fairness, we incorporate language and gender adapters along with adversarial invariance learning at the representation level. Beyond predictive performance, our framework is designed to support fine-grained interpretability, enabling analysis of feature importance, social roles, interaction strengths, and expert routing decisions. We provide detailed evaluations by analyzing the self and social branches independently. We further examine cross-group generalization across gender and language, and interpret social attention patterns underlying engagement predictions. Evaluations on NoXi-Base, NoXi-Additional, and NoXi-J demonstrate that our approach outperforms baselines by up to 64.1% while offering interpretable insights into engagement dynamics across language and gender groups.