Beyond Isolated Clips: Ambiguity-Aware Global Memory for Video Social Relationship Recognition
Abstract
Video-based social relationship recognition aims to identify the social relationships between characters, which can facilitate the construction of the multimodal knowledge graph and downstream video understanding tasks. Existing methods mainly focus on multimodal fusion and social graph modeling. However, a fundamental limitation of these methods is that they conduct relationship recognition on a single isolated clip, neglecting the rich historical context. Social relationships are inherently complex, information restricted to a single short clip is often insufficient to determine the relationship uniquely. To bridge this gap, we propose the Ambiguity-Aware Memory Network (AAMN), a framework that extends reasoning beyond the current clip by adaptively retrieving historical evidence when local inference is ambiguous. Specifically, we utilize the Jensen-Shannon Divergence to quantify the inconsistency between modalities as an indicator of ambiguity, which also serves as a trigger for memory activation. Furthermore, to handle the computational challenge of long videos, we construct a character-centric compressed memory bank that efficiently stores fine-grained spatiotemporal tokens. Extensive experiments on benchmarks demonstrate that AAMN achieves state-of-the-art performance, confirming the role of global context in reasoning complex social relationships. Our code is available at https://github.com/BoWang222/AAMN.