Joint Attention Modeling in Naturalistic Interactions with Social Gaze and Proximity Cues

Abstract

Joint attention (JA) is a core mechanism of social interaction, supporting coordination, knowledge transmission and early language development; deficits in JA are also a key early indicator of autism. While prior computer vision work often operationalizes JA as frame-based co-attention to a shared target, cognitive science defines it more precisely as a shared attentional state involving mutual awareness between interactants. This richer view encompasses diverse social behaviors, including brief disengagement from the shared focus, which are not captured by existing co-attention-based approaches. Despite its central role in social and developmental processes, automatic estimation of JA from multimodal visual data remains underexplored, particularly in naturalistic settings characterized by unstructured interactions and complex behavior. To address this gap, we propose a model for inferring joint attention directly from video in naturalistic environments. Our method follows a three-step pipeline: (i) we extract pose- and gaze-related perceptual cues to characterize participants and their visual focus; (ii) we derive relational features capturing proximity and shared attention dynamics, and (iii) we use temporal deep learning models to capture interactions over time and predict JA (see Figure 1). Experiments on a corpus of naturalistic home interactions involving adults and young children show that joint attention can be inferred from video in unstructured settings, with hand-related proximity and attention features emerging as particularly informative. These findings support the importance of modeling relational and interactional dynamics beyond simple co-attention, and highlight the potential of the proposed method for scalable, ecologically valid analysis of joint attention, with applications in research on language acquisition and child development.