From Detection to Mechanism Analysis: Interpretable Multimodal Concept Interactions in Gesture Pragmatics

Abstract

Co-speech gestures serve essential pragmatic functions in human communication, yet current computational efforts predominantly prioritize black-box detection or generation quality over mechanistic transparency. As a result, the cross-modal organization of these functions remains largely unexplored. In this paper, we shift the analytical focus toward mechanism-oriented analysis by examining how gesture, audio, and facial cues interact to construct pragmatic meaning. We introduce the Pragmatics-aware Structured Concept Interaction Model (PSCIM), an event-level framework that represents multimodal signals as interpretable concept families and explicitly models their structured interactions. Our findings show that pragmatic functions do not share a single globally optimal interaction channel; instead, they exhibit distinct class-dependent channel specialization across modalities. Furthermore, we identify representative interaction geometries characterized by the joint distribution of cross-modal concept pairs in a standardized feature space. These patterns reveal distinct cross-modal interaction signatures, ranging from coordinated concept-pair configurations, in which both concepts tend to be jointly elevated, to asymmetric configurations, in which one concept is relatively stronger while the other remains weaker. Comparisons with nonlinear glassbox models and cross-story analyses provide converging evidence that part of these structures is reproducible across modeling assumptions and the two narrative contexts in the corpus. This work offers a transparent framework for analyzing the multimodal organization of gesture pragmatics and provides a basis for future pragmatically grounded interactive systems.