CCPL: cross-modal complementary prototype self-learning for weakly-supervised temporal action localization

Abstract

Weakly-supervised temporal action localization aims to localize action instances in untrimmed videos using only video-level annotations. However, due to the limited representational capacity of single-modal information and the absence of temporal annotations, the localization framework struggles to extract representative action semantics, thereby restricting its temporal localization performance. To address this limitation, we propose a novel cross-modal complementary prototype self-learning framework that introduces supplementary textual information to guide the model in capturing complete action information. Building upon a visual modality localizer, we construct a complementary branch to learn secondary action semantics. Furthermore, we propose complementary prototype learning, which autonomously leverages dual-branch predictions to construct and update cross-modal action prototypes. By modeling similarity relationships between prototypes and visual features, the framework predicts complementary action sequences and ultimately performs action instance localization using multi-branch prediction sequences. Experimental results on two benchmark datasets demonstrate the effectiveness of the proposed method, outperforming existing superior approaches.