Aligning Gaze, Space, and Cognition: A Multimodal Triangulation Framework for Visitor Interactions in Cultural Heritage (An Expert vs. Non-Expert Comparative Study)

Abstract

Cultural heritage, as a combination of physical space and cultural significance, relies heavily on a deep understanding of visitor interaction patterns for its active conservation and public education. However, existing research is largely confined to laboratory or virtual environments, failing to capture the embodied interactions of visitors during real-world exploration. Furthermore, there is a methodological gap in multimodal spatio-temporal alignment and cross-modal synthesis across diverse knowledge backgrounds, limiting a holistic understanding of visitor experience. To address these challenges, the main novelty of the study is presenting a multimodal spatio-temporal alignment framework for analyzing how knowledge backgrounds influence human-environment interaction patterns. First-time visitors (N = 30; 12 visitors with an architectural background, hereafter “experts”, and 18 visitors without a formal architectural background, hereafter “non-experts”) were recruited for a real-world experiment in the Qianlong Garden. We integrated and associated four data modalities: GPS, on-site photography, eye tracking, and interview transcripts. Spatio-temporal patterns of visitor interactions were identified using cluster analysis; interaction scene selections were quantified through key event identification and semantic segmentation modeling; and information processing strategies were examined using LDA topic modeling. Results indicate both consistencies and significant differences in interaction patterns between the two groups. In the inter-group comparison, elements with high visual saliency functioned as common interaction anchors for both groups. However, the expert group exhibited more proactive, goal-oriented exploration patterns and demonstrated stronger sensitivity to culturally significant elements. Intra-group analysis further suggested more consistent interaction patterns within the expert group, whereas the non-expert group displayed higher dispersion. By identifying visual anchors and knowledge-dependent interaction patterns in cultural heritage exploration, this study provides exploratory evidence for spatial interventions in adaptive reuse, with important implications for enhancing public science education and the intelligibility of cultural heritage.