Confidence-Aware Semi-Supervised Vision–Language Contrastive Learning for Abnormal Behavior Recognition

Abstract

Reliable abnormal behavior recognition from surveillance videos is hindered by the high cost of clip-level annotation, the scarcity of abnormal samples, and the context-dependent nature of behavioral semantics. Although vision–language models offer strong semantic transferability, their application under limited supervision remains susceptible to noisy pseudo-labels and confirmation bias. We propose confidence-aware semi-supervised vision–language contrastive learning (CA-VLC), which jointly exploits limited labeled videos and abundant unlabeled videos. Building on an existing CLIP-initialized temporal backbone, CA-VLC combines behavior-only and context-enriched text prototypes through confidence- and agreement-guided semantic fusion. For unlabeled videos, the model generates predictions from weakly augmented views and selects reliable pseudo-labels using entropy-based confidence estimation and class-adaptive thresholds. Detached weak-view targets then supervise strongly augmented views through confidence-weighted self-training without requiring an additional teacher network. Furthermore, cross-view consistency regularization and confidence-aware contextual alignment suppress unreliable semantic cues and improve robustness to contextual noise. Experiments on CABR50 demonstrate consistent improvements across multiple labeled-data ratios, while evaluations on CABRZ6 and UCF-101 assess prompt-based transfer to predefined target label sets without target-domain fine-tuning. With 10% labeled videos, CA-VLC achieves 84.06% Top-1 accuracy and 83.51% Macro-F1, retaining 95.47% of its fully supervised Top-1 accuracy of 88.05%, thereby demonstrating its effectiveness for label-efficient abnormal behavior recognition.