Jul 2026· IEEE Transactions on Neural Networks and Learning Systems· Vol PP· 0 citations
Medicine
TL;DR
This work proposes VCF-CLIP, a visual context-driven fine-grained prompt learning framework built upon CLIP, and proposes the prompt prototype learning (PPL) strategy, which learns a pair of unified prompt prototypes representing general normal and anomalous states in a loss-guided manner, thereby eliminating the need for manual prompt design.
Abstract
Benefiting from recent advances in vision-language models (VLMs), numerous CLIP-based zero-shot anomaly detection (ZSAD) methods have been proposed to address the cold-start problem. Despite their impressive performance, these methods still depend on manual prompt engineering, and their coarse-grained text prompts struggle to capture the diverse patterns of anomalies, resulting in suboptimal visual-text alignment. To overcome these limitations, we propose VCF-CLIP, a visual context-driven fine-grained prompt learning framework built upon CLIP. The novelties of VCF-CLIP lie in two main aspects. First, we propose the prompt prototype learning (PPL) strategy, which learns a pair of unified prompt prototypes representing general normal and anomalous states in a loss-guided manner, thereby eliminating the need for manual prompt design. Second, we propose a lightweight prompt refinement adapter that dynamically aggregates multiscale and multilevel visual features to iteratively refine the prompt prototypes, enabling the generation of instance-specific prompts enriched with fine-grained information. We conduct extensive experiments on 14 benchmarks across industrial and medical domains, and show that VCF-CLIP outperforms existing state-of-the-art ZSAD methods.
The AP-LCA approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions.
Si-Ying Wu, Song Wu· International Conference on...· 0 citations
Vision-language models (VLMs), such as contrastive language-image pre-training (CLIP), exhibit powerful zero-shot generalization capabilities. Parameter-efficient fine-tuning (PEFT) techniques, notably prompt learning, have been extensively explored to adapt these models to downstream tasks. However, their efficacy remains constrained when transferred to specialized domains like remote sensing. We argue that the bottleneck stems not merely from the limited parameters of prompts, but essentially from the disruption of the input’s original image–text features and the lack of deep cross-modal alignment. In particular, existing methods typically rely on global attention or coarse-grained feature mapping. This inadvertently corrupts the original input representations, thereby impairing the model’s inherent generalization. Furthermore, their isolated unimodal gradient updates fail to bridge the semantic gap inherent in complex remote sensing scenes. To address these challenges, we propose tokenwise prompt-free learning (Tiper), shifting the optimization paradigm from introducing external prompts to precisely recalibrating the critical tokens that govern classification outputs. In particular, Tiper employs a hierarchical learner to supersede global prompts. Crucially, this learner intervenes exclusively on the specific core tokens (i.e., the CLS token in the visual branch and the EOT token in the textual branch), leaving other original input representations unperturbed. This fine-grained strategy effectively balances domain adaptation with the preservation of inherent generalization. Finally, we design the learner as a cross-modal coupled bridge with shared weights, enabling it to synchronously receive gradient feedback from both modalities and fostering profound multimodal collaboration. Extensive experiments validate our method on eight public remote sensing datasets covering diverse scenes and resolutions. In the base-to-new generalization task, Tiper outperforms the strong baseline MaPLe with a significant 3.7% improvement in the harmonic mean (HM). Notably, without relying on any external large-scale domain models, Tiper surpasses the latest domain-specific prompt learning methods (e.g., domain-controlled prompt learning (DCPL), domain prompt learning with quaternion networks (DPLQ)), demonstrating its superior adaptability for remote sensing image scene classification.
Tengfei Gong, Jun-Lin Wu, Yaxioong Chen et al.· IEEE Transactions on Geoscie...· 0 citations
The potential of vision-language models (VLMs) such as CLIP for zero-shot anomaly detection (ZSAD) is constrained by an inherent semantic-localization dichotomy. While CLIP’s global features excel at image-level classification, they lack the spatial sensitivity required for pixel-level segmentation. Existing approaches attempt to alleviate this issue through prompt optimization, which introduces a trade-off between global semantic discrimination and local localization accuracy, thereby limiting cross-domain generalization capability. To resolve this, we introduce HD-CLIP, a framework that decouples these competing objectives. HD-CLIP employs dedicated pathways for classification and localization, guided by a hierarchical and dynamic prompting mechanism that provides multilevel, content-adaptive cues. A novel localization-distilled supervision (LDS) loss then unifies these pathways by creating a probabilistic bridge between spatial evidence and semantic judgment. Extensive experiments on 14 challenging datasets achieve state-of-the-art performance, confirming that HD-CLIP effectively bridges the semantic-localization divide and advances the potential of VLMs for robust ZSAD applications.
Jielin Jiang, Yunxiang Chang, Hao Yin et al.· IEEE Transactions on Instrum...· 0 citations
Zero-shot industrial anomaly detection (ZIAD) aims to develop a unified model capable of directly identifying unseen anomaly categories in images without requiring reference samples. Recently, large-scale Vision-Language Models (VLMs) such as CLIP have shown great potential for solving this task. However, existing methods typically rely on manual text prompts to guide VLMs in anomaly detection, which often fail to capture fine-grained semantic cues, leading to limited accuracy. To address the challenge, this paper proposes a novel Cross-Modal Guidance Learning (CMGL) framework for ZIAD. Instead of handcrafted textual prompts, CMGL introduces learnable prompting mechanism to fully exploit the collaborative guidance between visual and textual modalities for efficient unseen anomaly detection. Leveraging the frozen image encoder of pre-trained CLIP, the CMGL extracts multi-scale patch tokens and global tokens of the input image as visual representations. Then, informed by the cross-modal information, adaptive prompt vectors are constructed to obtain textual representations. In the process, a Learnable Context Block (LCBlock) and a Multi-Layer Perceptron (MLP) are introduced to extract holistic semantics and fine-grained details, and an Adaptive State Vector Module (ASVM) is designed to learn generalized normal and abnormal state vectors from extensive text descriptions. By aggregating the outputs of these components, textual representations of the image are acquired through the frozen text encoder. Finally, a Local-Global Token Integrator (LGTI) and an Uncertainty-Aware Anomaly Fusion Module (UAFM) are proposed to achieve anomaly recognition and localization through visual–textual alignment. Extensive experiments on multiple industrial datasets demonstrate the superiority of our method. Note to Practitioners—This paper presents a Cross-Modal Guidance Learning (CMGL) framework to address anomaly detection under the zero-shot setting. Unlike previous approaches that rely on manually crafted text prompts, the proposed CMGL derives task-relevant prompt cues from cross-modal data by the designed learnable prompting mechanism, guiding the model to automatically recognize and localize unseen anomaly categories without requiring any reference samples. Extensive experiments demonstrate the effectiveness and strong generalization capability of the proposed approach. Benefiting from these properties, our method provides a novel and effective ZIAD solution for identifying potential anomalies in real-world industrial scenarios where data distributions are uncertain or anomaly-related information cannot be clearly specified. Our project page is publicly available at https://aicoder12.github.io/CMGL/
Tiyu Fang, Lin Zhang, Ran Song et al.· IEEE Transactions on Automat...· 0 citations
Zero-shot anomaly detection (ZSAD) aims to detect and localize anomalies in unseen categories without access to target-specific training data. Although recent CLIP-based methods have demonstrated promising generalization through vision-language alignment, they remain limited in capturing diverse anomaly semantics and subtle local variations. To address these limitations, we propose VFAD, a unified framework that combines variational semantic prompting with frequency-adaptive representation learning. Specifically, we introduce a Variational Semantic Prompt Extractor (VSPE), which adaptively aggregates anomaly-relevant local semantics from dense patch tokens and regularizes them through a variational information bottleneck, thereby incorporating fine-grained visual cues and enabling more precise cross-modal alignment. Furthermore, we develop a Frequency-Adaptive Representation Aggregation (FARA) module that leverages wavelet-based frequency decomposition and frequency-specific expert aggregation to enhance anomaly-discriminative visual representations. By jointly strengthening semantic guidance and visual representation learning, VFAD improves both anomaly discrimination and fine-grained localization. Extensive experiments on 13 industrial and medical benchmarks demonstrate that VFAD consistently outperforms existing state-of-the-art ZSAD methods across diverse anomaly scenarios. The code will be publicly available upon publication.
Zero-shot anomaly detection (ZSAD) aims to identify anomalies in target datasets without accessing their samples. Although CLIP and other large-scale vision language models show strong generalization, their potential for multi-level feature extraction in ZSAD remains underexplored. To address this, we propose a Multi-Level Feature Learning (MLFL) framework to enhance the zero-shot capability of CLIP via hierarchical alignment. MLFL adopts a two-stage training paradigm: Multi-Level Text Prompt Tuning (MLTP) and Multi-Level Text-Image Feature Alignment (MLFA). MLTP learns object-agnostic and object-aware prompts tailored to different encoder blocks. MLFA aligns textual and visual features using linear layers for shallow blocks and a Deep Feature Alignment (DFA) module for deep blocks. To compress parameters and preserve semantics, we introduce a Generalized Prompt Distillation (GPD) module that distills object-aware prompts into a unified representation. Experiments on seven industrial datasets achieve state-of-the-art performance, and deployment tests on edge devices demonstrate the potential applicability of the framework in practical industrial scenarios.
Jianfeng Qiu, Junfa Li, Juan Xie et al.· Complex & Intelligent Sy...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
