Hierarchical Contrastive Multimodal Alignment for Contactless Sensing: An Empirical Study on Zero-shot mmWave-based HAR

Abstract

Human Activity Recognition (HAR) enables intelligent systems to identify human actions using modalities like cameras, wearables, and radar. This capability is critical in healthcare, providing the automated, continuous supervision necessary to improve patient safety, elderly care, and emergency response. Within this domain, millimeter-wave (mmWave) radar has emerged as a particularly attractive modality. Its privacy-preserving nature and low-light robustness make it ideal for non-intrusive monitoring in sensitive settings without compromising user dignity. Recent frameworks, such as mmCLIP, have explored zero-shot generalization for mmWave-based HAR by adapting contrastive language–image pretraining (CLIP). By aligning mmWave signals within CLIP’s shared vision–text embedding space, these frameworks inherit its broad semantic generalizability. However, such approaches rely exclusively on instance-level contrastive alignment, thereby overlooking broader categorical structures. In this paper, building on the mmCLIP training framework for CLIP-based HAR, we introduce a staged training scheme that first aligns mmWave signals with text at the prototype level to establish coarse categorical semantics, and subsequently refines the alignment at the instance level, enabling the model to commit to stable categorical structure before refining intra-class distinctions. In addition, we further investigate the application of Elastic Weight Consolidation (EWC) to empirically evaluate whether strictly preventing the instance-level objective from overwriting the initial prototype-level semantic information benefits overall model performance. Extensive evaluation demonstrates that our proposed staged training scheme outperforms the baseline approach on zero-shot classification of real-world mmWave data, though future research may be needed to help confirm its efficacy. Additionally, our investigation into EWC reveals that the prototype objective operates most effectively when utilized as a flexible semantic prior rather than a strict constraint.