A novel MLIR-PL model with Visual–Label Alignment and Attribute-Aware Prompt and Attribute-Aware Prompt, which sufficiently harnesses the capabilities of large-scale pre-trained vision-language models is proposed, achieving the best performance comprehensively.
Abstract
The problem of Multi-Label Image Recognition with Partial Labels (MLIR-PL) is a significant challenge in computer vision, primarily due to the scarcity and high cost of complete annotations. Recent advances have leveraged large-scale vision-language models, such as CLIP, to establish rich correspondences between images and their labels, thereby improving the MLIR-PL performance. However, the existing CLIP-based methods have not fully exploited fine-grained local image features to mitigate interference from semantically irrelevant regions. Moreover, many studies have oversimplified the use of prompt contexts, limiting their ability to comprehensively capture the multi-dimensional attributes of categories. To address these limitations, this article proposes a novel MLIR-PL model with Visual–Label Alignment and Attribute-Aware Prompt (VA \({}^{3}\) P), which sufficiently harnesses the capabilities of large-scale pre-trained vision-language models. In the model, we design a Visual–Label Alignment module to establish a mapping between local image features and category text representations, conspicuously reducing the interference from irrelevant regions. Additionally, our Attribute-Aware Prompt module offers diverse contextual information, providing a more comprehensive representation of the category’s attributes. Extensive experimental results on the COCO 2014 and VOC 2007 datasets, compared with multiple state-of-the-art methods, demonstrate that our model achieves the best performance comprehensively, verifying the advantages of the proposed model in the MLIR-PL task.
The AP-LCA approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions.
Si-Ying Wu, Song Wu· International Conference on...· 0 citations
In semi-supervised object detection (SSOD), due to the limited availability of labeled data, the quality and quantity of pseudo labels generated from unlabeled images are crucial for model training. Our study reveals that in the early stages of training, the number of usable pseudo labels is very low, which hampers the model's ability to converge quickly. Furthermore, throughout the training process, even though pseudo-labels are generated with high confidence, a significant number of labeling errors still occur. To address these challenges, we propose a novel framework, Generation-to-Optimization of Pseudo Labels (GO-PL), which comprehensively improves the model's performance in both the pseudo labels generation stage and the pseudo labels optimization stage. Specifically, we design a CLIP-based Pseudo Label Generation (CPLG) module, which matches proposals from the teacher model with CLIP text embeddings to generate semantically consistent high-confidence pseudo labels. These labels are further integrated into student training via an auxiliary loss, enabling early-stage guidance with stable and informative supervision signals. We also introduce Pixel-guided Instance Weighting (PIW) module that utilizes the SAM to assess the pixel-level quality of pseudo labels generated by the teacher model. Combined with a local voting mechanism, PIW dynamically assigns weights to each pseudo-label, thereby optimizing the impact of labels with varying quality on gradient updates. GO-PL demonstrates strong performance on standard SSOD benchmarks, including the MS-COCO and PASCAL VOC datasets. Notably, it achieves 26.7% mAP with only 1% labeled data on the MS-COCO, surpassing the previous best approach by 1.7%.
Xi Yang, Penghui Li, Nannan Wang· IEEE Transactions on Image P...· 0 citations
Video topic recognition faces core challenges such as the limitations of static representations, cross-modal semantic misalignment, insufficient coverage of single-scale features, and weak temporal dynamic modeling. This paper finds that existing methods have significant deficiencies in local detail perception, leading to the neglect of key local cues such as "test tubes" in "laboratory" scenes, or the inability to simultaneously capture global environment and local details in "wedding" scenes. To address this, we propose a semantic-guided multi-scale feature pyramid learning method. The core innovation lies in the design of a temporal-semantic guided multi-scale local feature extractor (MLFE). This module can not only handle spatial multi-scale features but also incorporate temporal dynamics and textual semantic guidance to achieve adaptive scale selection. Based on this, we have constructed a complete recognition framework, including an improved cross-frame communication mechanism and a multi-granularity dynamic cue generation module. Experiments on benchmark datasets such as Kinetics-400 and Kinetics-600 show that our method significantly outperforms existing methods in scenarios requiring fine-grained local feature recognition, especially in topics such as "laboratory," "medical surgery," and "cooking process.
Jin-Yao Zhang· Poster Volume 0007 The 2026...· 0 citations
Image-text matching faces the significant challenge in effectively mitigating the large visual-semantic discrepancy among different modalities. Existing studies mainly address this by projecting multi-modal data into a common subspace for measuring their semantic similarities. However, most of them often overlook the inherent asymmetry for directional associations, i.e., the differences between im-age-to-text and text-to-image affinities, which often results in inaccurate retrieval results. To tackle the challenge, we propose a Dual Associations Semantic En-hancement (DASE) model to capture bidirectional image-text semantic associa-tions. Specifically, we first build a two-layer GCN fusion network to construct and mine the semantic associations for each modality. And then, due to the inher-ent asymmetry of directional associations, a Dual Associations Alignment Mod-ule (DAAM) is designed to capture the dual associations between visual and tex-tual modalities, enabling comprehensive cross-modal fine-grained interaction. Fi-nally, global alignment is incorporated with the local alignment to achieve full semantic matching across heterogeneous modalities in a unified embedding space. Experimental results on two publicly available datasets demonstrate that the pro-posed DASE model achieves significant performance improvements in image-text matching tasks compared to baseline methods, validating its effectiveness and superiority.
Xinlin Zhao· Poster Volume 0008 The 2026...· 0 citations
This work proposes an integrated learning paradigm that simultaneously enhances feature compactness and improves robustness against label noise and introduces a feature disentanglement mechanism that isolates reliable label-related feature representations from spurious ones introduced by noisy supervision.
Yuzhi Tao, Anhui Tan· Computers, Materials & C...· 0 citations
Recent studies develop pixel-level multimodal large language models (MLLMs) that support both Region Segmentation and Region Understanding, extending multimodal interaction from whole images to specific objects and regions. However, these methods face two fundamental challenges. First, the scarcity of high-quality mask--text pairs leaves abundant mask annotations without corresponding language supervision. Second, discrepancies in supervision formats and learning-signal densities induce optimization interference between Region Segmentation and Region Understanding. To address these challenges, we propose PixVL, a self-supervised post-training framework that introduces a unified Mask--Text Consistency Cycle, enabling pixel-level MLLMs to generate and self-verify regional descriptions and learn from unlabeled data. We found that direct cycle based solely on geometric reconstruction is unreliable because re-segmentation IoU does not faithfully reflect the semantic quality and referring sufficiency. PixVL therefore introduces confuser-aware semantic verification, which uses the model's confidence when it correctly chooses the target among highly similar candidate regions, and assigns zero reward to an incorrect choice. Meanwhile, PixVL performs cross-view verification using temporally separated video frames or geometrically transformed image views, preventing cyclic learning from collapsing to positional and shape shortcuts. Finally, a quality-coupled bidirectional learning strategy uses the highest-reward description to guide Text-to-Mask learning. This strategy transforms Region Understanding and Region Segmentation from competing tasks into mutual generators and verifiers. Experiments demonstrate that PixVL improves both region understanding task and segmentation task.
Yicheng Xiao, Haoxuan Ma, Caorui Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
