Jul 2026· International Conference on Computer Vision, Al and Intelligent Automation· Vol 14260, pp. 1426003 - 1426003-7· 0 citations· 17 references
Engineering
TL;DR
The AP-LCA approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions.
Abstract
Pre-trained vision-language models (VLMs) like CLIP have achieved remarkable success in zero-shot visual recognition. While recent advancements leverage Large Language Models (LLMs) to generate fine-grained category descriptions in order to enhance CLIP-based models, they often suffer from significant spatial granularity mismatch (fine-grained category descriptions vs. the global image) and rely heavily on labor-intensive handcrafted prompt templates. To address these challenges, we propose the Adaptive Prompt-guided Local Cross-modal Alignment (AP-LCA) approach. Our approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions. Additionally, we incorporate an in-context learning (ICL) mechanism to automate the generation of task-adaptive prompts, seamlessly evolving from manual templates to context-aware descriptions that capture diverse visual concepts. Extensive experiments across eight benchmark datasets demonstrate the clear superiority of our proposed method.
This work proposes VCF-CLIP, a visual context-driven fine-grained prompt learning framework built upon CLIP, and proposes the prompt prototype learning (PPL) strategy, which learns a pair of unified prompt prototypes representing general normal and anomalous states in a loss-guided manner, thereby eliminating the need for manual prompt design.
Kaiwen Fu, Fei Qi, Chengyuan Chang et al.· IEEE Transactions on Neural...· 0 citations
Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and Attention Refined Segmentation for CLIP), a novel framework for zero-shot semantic segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-attention block to accurately restore object boundaries. A set of experiments on six public datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.
Nagito Saito, Shintaro Ito, Koichi Ito et al.· International Conference on...· 0 citations
Generalized zero-shot learning (GZSL) addresses the challenging task of recognizing both seen and unseen classes by leveraging shared semantic knowledge. A core challenge in this domain is achieving robust visual-semantic alignment to transfer knowledge from seen classes to novel classes. Current state-of-the-art methods typically fine-tune large-scale visual backbones on scarce training data. However, this approach frequently leads to severe overfitting to seen classes, which significantly degrades performance on novel categories. To mitigate this issue, we propose the Dual Adaptive Visual-Semantic Prompt Collaboration Network (VSPCN+), a novel framework that utilizes prompt-tuning for effective feature adaptation. Our method introduces a dual-prompt mechanism comprising both visual and semantic prompts. The semantic prompts guide the visual encoder to learn visual features that are more semantically consistent with class attributes, while the visual prompts steer the semantic encoder to generate semantic representations that are more visually grounded. This collaborative process enhances the overall visual-semantic consistency. A key innovation of our work is the dynamic generation of instance-adaptive prompts, which contrasts with existing prompt-learning methods that rely on static, global prompts. By tailoring prompts to individual instances, our approach enhances the model’s robustness and generalization capabilities across diverse visual inputs. This collaborative adaptation, guided by our dual-prompt mechanism, allows the visual and semantic encoders to produce consistent representations for effective visual-semantic alignment. Extensive experiments on standard GZSL benchmarks demonstrate that our proposed VSPCN+ performs favorably against several state-of-the-art methods.
Huajie Jiang, Zheng-Xian Li, Yuankai Qi et al.· International Journal of Com...· 0 citations
A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which explicitly injects positional cues into the compressed representations and strengthens spatial structure.
Yuchen Sha, Lingli Wan, Ge Yang et al.· The Visual Computer· 0 citations
Adapting CLIP for zero-shot sketch-based image retrieval (ZS-SBIR) via prompt learning faces a fundamental tension: the model must bridge the sketch-photo domain gap through task-specific adaptation, yet the added flexibility risks overfitting to seen training categories and eroding CLIP's zero-shot generalization. We present SeCo-SBIR, a semantically consistent prompt learning framework that resolves this tension from both sides. First, a text-guided multi-modal prompting strategy routes learnable prompt vectors through CLIP's text encoder and projects the resulting intermediate representations into the visual encoder at every layer via learnable coupling functions. Because the text encoder has already learned robust, abstract category-level semantics from large-scale language supervision, this mechanism injects transferable semantic knowledge directly into the visual pathway - adapting the model to the sketch-photo domain while inherently favoring generalization to unseen classes. Second, a perturbation-based consistency constraint addresses the residual overfitting risk from the learnable coupling functions by aligning the adapted model with a frozen CLIP reference branch using an asymmetric InfoNCE objective - augmented inputs feed the frozen branch while clean inputs feed the trainable branch - anchoring the learned representations to CLIP's generalizable feature space. Together with lightweight adapters and a multi-objective loss combining triplet, NT-Xent, and classification terms, SeCo-SBIR achieves state-of-the-art results on all three standard ZS-SBIR benchmarks across categorical, generalized, and across-dataset settings.
L. Dang, Tuan Nguyen Huu, Hieu Nguyen et al.· 0 citations
Fine-grained video action recognition remains challenging because action categories often differ only in subtle inter-class variations and complex temporal dynamics. Recent Contrastive Language–Image Pre-training (CLIP)-based extensions perform well on general action recognition, but they typically rely on early global pooling of video features. Such coarse representations discard the fine temporal cues that distinguish subtle actions, causing a granularity mismatch in cross-modal alignment. To address this, we propose Structure-Aware Semantic-Adaptive (SASA)-CLIP, a framework for multi-granular cross-modal alignment. SASA-CLIP adopts a dual-branch design: a coarse-grained branch captures the global context, while a fine-grained branch matches descriptors against individual frames before aggregation, rather than pooling features early. To keep this alignment temporally coherent, we introduce a Gaussian prior as a temporal structural constraint, encoding the inductive bias of local temporal continuity into the attention matrix to guide an ordered alignment of key action segments along the temporal axis. On Kinetics-400 (ViT-B/32), SASA-CLIP reaches a Top-1 accuracy of 81.37%, improving over the X-CLIP baseline by 0.97%; on HMDB-51 and UCF-101 (ViT-B/16), it reaches 74.0% and 96.81%, improving by 3.25% and 2.61%, respectively. It also transfers to the zero-shot setting, improving over the baseline on HMDB-51 and UCF-101. These results show that combining multi-granular representations with a temporal structural prior benefits fine-grained recognition, suggesting that SASA-CLIP is a practical option for real-world visual sensing applications such as intelligent surveillance and wearable activity monitoring.
Xiaowei Han, Wenbao Si, Honghui Zhang et al.· Italian National Conference...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
