Modeling Participant Behavior Across Utterances and Silent Intervals for Engagement Estimation of Individuals with ASD
TL;DR
A time-based input format that preserves behavioral features during silent intervals and aggregation over time-series data for estimating the engagement of individuals with ASD is proposed.
Abstract
Engagement estimation for individuals with Autism Spectrum Disorder (ASD) is an important step toward supporting communication between ASD and typically developing (TD) individuals in group conversations. However, existing methods rely on utterance-based representations, which discard important nonverbal cues conveyed during silent intervals: periods in which no one is speaking. In this study, we propose a time-based input format that preserves these intervals. We further propose a two-stage hierarchical engagement estimation model that leverages learnable CLS embeddings: the first stage aggregates the time-series features of each participant’s each modality (face, audio, and language), and the second stage jointly encodes the relationships across modalities and across participants. Experimental results show that our proposed model achieves the highest Macro-F1 of 53.7% among the compared approaches. These findings demonstrate the effectiveness of behavioral features during silent intervals and aggregation over time-series data for estimating the engagement of individuals with ASD.