Human–computer interactive video summarization system driven by visual scanpaths generation for long video understanding

Abstract

Abstract. Video summarization plays a crucial role in efficient long video understanding and question answering. However, existing deep-learning-based approaches suffer from two major limitations. On the one hand, they lack interpretability, often leading to a mismatch between the generated summaries and users’ actual interests. On the other hand, as video sequences grow longer, conventional transformer architectures struggle to maintain long-range contextual dependencies while remaining computationally efficient. To address these challenges, we propose a human–computer interactive video summarization system driven by visual scanpaths. First, we design a visual-question-answering-based scanpath generator that simulates human gaze behavior in a task-oriented manner, effectively filtering out irrelevant content. Second, based on the generated scan paths, we develop an adaptive frame prioritization strategy to extract top-K keyframes consistent with human attention and interest perception. To efficiently preserve the contextual dependencies among these distant keyframes, we further introduce a token compression module, enabling compact visual representations and efficient information propagation with fewer tokens. Finally, we integrate a user-centric, interactive visual question answering system that dynamically customizes video summaries according to user queries. Extensive experiments on multiple datasets demonstrate that our interactive, gaze-driven summaries align better with user preferences than conventional methods.