Skip to content

MODdapter: Spatially-aware text embeddings for zero-shot semantic segmentation.

Sep 2026 · Neural Networks · Vol 206 Pt A, pp. 109694 · 0 citations · 67 references
Medicine

TL;DR

MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition of unseen objects.

Abstract

Vision-language models show promise in zero-shot semantic segmentation, but a key challenge is the disconnect between text and visual features. While text embeddings can roughly localize unseen objects, they often lack the fine-grained detail necessary for accurate segmentation, leading to oversegmentation or undersegmentation. To address this, we propose MODdapter, a Multimodal Adapter with Proportional-Integral-Derivative (PID) control for precise dual-space alignment of vision and language features. Our core idea is to leverage visual cues obtained during testing to complement textual information, improving the alignment between text and visual features. MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition. To maintain stability and counteract potential oscillations caused by deviations in the extracted instance cues, we incorporate a PID control module that regulates the alignment process. PID control module leverages the combined joint constraints of short-term (derivative) and long-term (integral) memory, achieving fine-grained recognition of unseen target. This strategy significantly enhances the recognition of unseen objects, achieving state-of-the-art performance across multiple datasets. On the challenging COCO-20i dataset, our method outperforms the current state-of-the-art by a significant improvements of 6.9%.

View source

Similar papers

Preprint Sep 2026

ViCo-SAM3: Vision-Conditioned Alignment for Open-Vocabulary Camouflaged Object Segmentation

This work proposes ViCo-SAM3, a Vision-Conditioned alignment framework designed for OVCOS, which introduces vision-conditioned (ViCo) module, which dynamically modulates text embeddings with global visual context, enabling textual representations to adapt to the current image content and thereby effectively bridges the...

Qiang-Qiang Zhou, Wengang Tang, Yong Chen et al. · 1 citation
Preprint Oct 2026

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance re...

Tian-Qi Wei, Zhi Chen, Yu Xin et al. · 0 citations
Conference Open access Sep 2026

Auxiliary text-guided image restoration for image-text matching

A new ITM framework that improves the model's discriminative performance by focusing on localized core attributes and has a better robustness in handling highly similar hard negatives, which provides a new way to address cross-modal hard sample discrimination.

Ya-Ting Liu, Kuang-Rong Hao · 0 citations
Preprint Oct 2026

ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference

Text-to-image models have achieved remarkable success in generating high-quality images from textual descriptions. However, they often struggle with"semantic drift,"where the generated output fails to precisely align with complex or nuanced text prompts. While recent approaches have attempted to address semantic errors...

К. М. Камали, Basim Azam, Tuan-Duong Trinh et al. · 0 citations
Preprint Oct 2026

TIE: Training-free CLIP Embedding Guidance for Compositional Text-to-Image Alignment

Text-to-image (T2I) diffusion models have advanced notably, but still fail to satisfy prompt conditions, resulting in attribute mismatches and missing objects. These errors normally fall into two categories: concept loss, when an object or attribute (i.e., concept) goes missing, and concept confusion, when an attribute...

Su-Jun Hong, Tae Eun Choi, Youngjun Jun et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.