MEMC: Masked modeling with efficient and minimal contrastive learning for self-supervised skeleton-based action recognition.

Abstract

Unsupervised 3D skeleton-based action recognition offers advantages in terms of robustness and computational efficiency. However, prevailing paradigms, such as masked skeleton modeling (MSM) and contrastive learning (CL), have inherent limitations: MSM often learns representations with limited discriminative capability, whereas CL may fail to adequately capture fine-grained structural information. Furthermore, integrating the two paradigms through multi-task learning (MTL) can lead to gradient conflicts that limit performance. To address these issues, we propose MEMC (Masked Modeling with Efficient and Minimal Contrastive Learning), a novel framework that adopts an efficient sequential cascade strategy based on layer-grafted pretraining. MEMC first uses MSM to learn low-level representations and then refines high-level representations through CL, thereby avoiding the gradient conflicts associated with MTL. To improve the discriminative capability of the learned representations, particularly for subtle actions, we introduce two CL enhancements. First, Topology Distance-Aware Chain Motion Modeling incorporates skeletal topology priors to capture discriminative motion patterns along skeletal chains. Second, Frequency Band-Aware Contrastive Learning with Frequency Band Pooling (FBP) separates and integrates high-frequency details with low-frequency global context. Extensive experiments on three benchmark datasets demonstrate the effectiveness of MEMC and show its superior performance compared with state-of-the-art methods.