A novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA.
Abstract
Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.
A Semantic Manifold-Aware Similarity Learning (SMSL) framework, where the term "semantic manifold" is used in an operational sense to denote a topology-aware organization of textual semantic units induced from token/entity interactions, rather than a strict low-dimensional differentiable manifold.
GenView is presented, a controllable framework that augments the diversity of positive views leveraging the power of pretrained generative models while preserving semantics, and an adaptive view generation method that dynamically adjusts the noise level in sampling to ensure the preservation of essential semantic meani...
Xiao-Jie Li, Yi-Bo Yang, Xiang-Tai Li et al.· 0 citations
Interestingly, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM, revealing that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.
Wei-Jian Jian, Xiao-Yue Zhang, Bin Xiao et al.· European Conference on Compu...· 0 citations
This work presents a novel self-supervised architecture centered on graph prototype learning that sets a new state-of-the-art on the ARMM dataset with an accuracy of 95.70%, substantiating the efficacy and transferability of prototype-guided self-supervised learning for skeleton-based action representation.
Zhijie Xu, Hong-Wei Chen, Xia Li· International Journal of Mac...· 0 citations
G2D is proposed, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image and transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning.
Zehua Hao, Fang Liu, Qinliang Wang et al.· 0 citations
A multimodal Graph-based Fusion (MGF) framework that learns visually grounded semantic representations to enhance FSOR performance and achieves superior open-set recognition and competitive closed-set classification performance.