PISA achieves state-of-the-art performance without requiring access to source-domain data, surpassing existing methods by 3.92% in AP@50% on COCO-C.
Abstract
Open-vocabulary object detection test-time adaptation (OVOD-TTA) aims to address the performance degradation that pre-trained base models suffer when encountering image-domain shifts. Existing source-free OVOD-TTA methods rely either on refined test-time information for re-scoring or on pseudo-labels for self-training, leading to significant accuracy degradation when initial predictions are poor. Meanwhile, most conventional source-domain estimation methods recover abstract, sparse representations suitable for the classification task, but fail to capture the dense, concrete features required for detection. To address these issues, we propose PISA, a novel source-free OVOD-TTA method that can be seamlessly integrated into open-vocabulary visual backbones. The core components of our method are the Corruption-Invariant Feature Extractor (CIFE), the Feature Alignment Module (FAM), and a multi-scale alignment framework (BAA). To capture detection-suitable features, we develop CIFE to exploit the invariance of CLIP's visual features across corrupted images, ensuring robustness against various corruptions. We further develop FAM and BAA for the pre-training and adaptation to transform the corruption-invariant features into pseudo-individual source-domain features that are close to the original source-domain features. In this way, dense and concrete pseudo-individual source-domain features are used for supervision instead of unreliable pseudo-label signals. Experiments on the corrupted VOC-C, COCO-C, and LVIS-C benchmarks across three base models demonstrate that PISA substantially improves both the localization precision and the category recognition accuracy of the original models. Notably, PISA achieves state-of-the-art performance without requiring access to source-domain data, surpassing existing methods by 3.92% in AP@50% on COCO-C.
Open-vocabulary object detection (OVD) models generally leverage vision-language pre-trained models to recognize novel categories via arbitrary text prompts. Nevertheless, their performance is restricted by two core limitations: semantic discontinuity arising from hard binary supervision in contrastive learning, and the inflexibility of single prompts to convey complex detection intents in practical scenarios. To tackle these issues, this paper proposes YOLO-TVP (Text–Visual Prompt), an efficient OVD framework built on the YOLO architecture, with two key designs. First, a Semantic-Target Soft Cross-Entropy (ST-SoftCE) loss is introduced. It constructs semantic target distributions from inter-class similarities in the shared text embedding space for open-vocabulary inference and supervises the detector's classification branch. This design embeds semantic relevance into supervision and enhances fine-grained discrimination among semantically similar categories. Second, a unified class-prompt embedding interface is developed to support both textual and visual prompts. Text prompts are projected into the shared prompt space, while visual prompts are formed by learnable weighted fusion of CLIP semantic priors and backbone visual features, eliminating the need for multi-modal prompt concatenation during inference. Experiments validate the effectiveness: on Flickr30k image-text retrieval with ResNet-50, ST-SoftCE improves Recall@1 by 9.56% over standard cross-entropy. For open-vocabulary detection trained on COCO+Flickr30k and evaluated on LVIS, ST-SoftCE delivers a 13.15% relative mAP
50
improvement to YOLO-World. With scratch-trained detector weights and frozen vision-language encoders used only for prompt-side
priors, YOLO-TVP achieves competitive prompt-conditioned detection while preserving real-time efficiency, verifying its advances in semantic alignment and prompt-adaptive detection.
Zhong-Hao Wang, Jun Huang, Yu Kang et al.· Intelligent Data Analysis· 0 citations
Open-vocabulary semantic segmentation (OVSS) repurposes a pretrained CLIP encoder for dense prediction without additional labeled supervision. Existing methods improve CLIP's spatial behavior either by redesigning its internal attention or by injecting features from auxiliary vision foundation models; both require access to the host's internal computation and are tailored to its specific forward pass. In this work, we propose Test-time Prototype Adaptation (TPA), a training-free plug-in that operates at the output level, leaving the host's forward pass and weights unmodified. By leveraging a lightweight transductive adaptation phase, TPA identifies confident anchor patches from the host's own output predictions on a small pool of unlabeled deployment-domain images, and aggregates their frozen DINO features into per-class prototypes; at inference, a single cosine similarity lookup against this frozen bank provides an auxiliary score fused linearly with the host's logits. TPA composes with five representative OVSS hosts spanning attention-redesign and VFM-injection designs, across three CLIP backbones, eight benchmarks, and multiple internal VFM choices. Under a single set of hyper-parameters and without per-host tuning or parameter updates, TPA consistently improves segmentation accuracy, with as few as approximately 10% of unlabeled deployment-domain images sufficing for effective bank construction on most benchmarks.
Haozhe Wang, Jintao Cheng, Weibin Li et al.· 1 citation
Source-free domain adaptation (SFDA) aims to adapt pre-trained source models to new target domains without requiring access to any source domain data, thereby addressing privacy and efficiency concerns. Existing SFDA methods for object detection primarily follow a teacher–student self-training paradigm; however, their performance is often limited by noisy pseudo-labels. To address this issue, this paper proposes an SFDA object detection framework for the YOLO family of single-stage detectors. First, a weak–strong pseudo-label consistency filtering strategy is designed to remove unreliable pseudo-labels by exploiting the prediction consistency across different augmented views. Second, a multiscale object-level contrastive learning mechanism is introduced to extract object-level features at multiple feature scales, thereby enhancing the consistency and discriminability of object representations across different views and scales through supervised contrastive constraints. Experimental results show that the proposed method consistently outperforms the baseline on multiple cross-domain detection tasks, demonstrating its effectiveness and good generalization ability under the source-free setting.
Open-vocabulary object detection leverages vision-language pre-training to achieve cross-category generalization. With the progressive refinement of models such as GLIP, Grounding DINO, and YOLO-World, detectors have exhibited substantially enhanced zero-shot recognition capabilities on open vocabularies, with YOLO-World further advancing this capability into an efficient single-stage detection paradigm. However, during scenario-specific fine-tuning, training that relies solely on ground-truth annotations of base classes induces rapid degradation in the model’s discriminative ability for novel classes, often reverting to closed-set detector behavior. To address this issue, the present paper proposes a forgetting-resistant fine-tuning framework that decouples the training objectives into strong supervised adaptation for base classes and weak supervised preservation for novel classes. Specifically, novel pseudo-boxes are generated once by a frozen zero-shot teacher before any task-specific fine-tuning and are refined through threshold filtering, score weighting, and soft-ignore mechanisms to reduce the impact of noisy pseudo-supervision. Subsequently, teacher-guided logit distillation is incorporated, constraining the student model exclusively at the output distribution level to preserve its response relationships to the novel vocabulary. Experimental results demonstrate that conventional fine-tuning on YOLO-World-v2-S reduces Novel AP50 from 57.6 to 1.6. In contrast, through stepwise integration of the freezing strategy, pseudo-labels, and logit distillation, the proposed method elevates Novel AP50 to 51.6 and achieves an All AP50 of 53.7. These findings show that the proposed approach substantially mitigates catastrophic forgetting while preserving the original single-stage inference pipeline.
PuRF is introduced, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models that consistently outperforms state-of-the-art methods on ViT-B/32 across five datasets.
Yiwen Liang, Hui Chen, Yizhe Xiong et al.· 0 citations
Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.
Jing Li, Pan Liu, Meng Zhao et al.· IEEE transactions on multime...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
