Skip to content

XMask3D++: Cross-Modal Mask Reasoning for Open Vocabulary 3D Segmentation.

Sep 2026 · IEEE Transactions on Pattern Analysis and Machine Intelligence · Vol PP · 0 citations
Medicine

TL;DR

This work developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks, and introduces a 2D-to-3D mask adaptation module to improve the quality of back-projected 3D masks.

Abstract

Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. To improve the quality of back-projected 3D masks, we further introduce a 2D-to-3D mask adaptation module that iteratively refines both 3D mask contours and condenses mask prototype features in the advanced XMask3D++ framework. The enhanced mask prototypes, which are also explicitly regularized by the mask loss, are then integrated into the fused features to boost open-vocabulary capacity. We demonstrate the effectiveness of XMask3D++ through consistent improvements across various benchmarks and datasets, covering both indoor and outdoor scenarios. We further conduct zero-shot and domain transfer experiments to validate the generalization capability of the proposed method. In addition, we evaluate the framework's flexibility by replacing the 3D geometric feature extractor and the 2D open-mask generator with more advanced models, demonstrating the scalability and adaptability of our approach.

View source

Similar papers

Preprint Sep 2026

ViCo-SAM3: Vision-Conditioned Alignment for Open-Vocabulary Camouflaged Object Segmentation

This work proposes ViCo-SAM3, a Vision-Conditioned alignment framework designed for OVCOS, which introduces vision-conditioned (ViCo) module, which dynamically modulates text embeddings with global visual context, enabling textual representations to adapt to the current image content and thereby effectively bridges the...

Qiang-Qiang Zhou, Wengang Tang, Yong Chen et al. · 1 citation
Aug 2026

PointPDF V2: A Unified Framework for Continual Open-World 3D Semantic Segmentation.

This work proposes PointPDF V2, a unified framework that integrates open-set recognition (OSS) and incremental learning (IL) into a cohesive pipeline and introduces a more challenging continual OWSS protocol in 3D, where models must simultaneously preserve the known-class performance, acquire new knowledge, and still i...

Jinfeng Xu, Xianzhi Li, Yi-Xue Hao et al. · 0 citations
Sep 2026

Hybrid Gaussians for Robust Open-Vocabulary 3D Segmentation with Multi-View Object Association and Boundary Refinement

This work introduces Hybrid Gaussians, a unified 3D representation jointly modeling object association and language-aligned semantics and its Multi-View Object Association mechanism combines Observation Fusion and Semantic Contrastive Learning to improve identity consistency and semantic discrimination.

Xue-Qi Qiu, Yue-Ming Sun, Tian-Yu Zhang et al. · 0 citations
Sep 2026

PePESeg3D: Perception Prior Enhances Multi-Scale Segmentation for 3D Gaussian Splatting

PePESeg3D achieves state-of-the-art performance in both multi-scale segmentation and scene reconstruction, highlighting the importance of integrating perception priors into both geometry optimization and feature learning for accurate multi-scale 3D segmentation.

Sung Choi, Seunghee Koh, Junmo Kim · 0 citations
Sep 2026

MODdapter: Spatially-aware text embeddings for zero-shot semantic segmentation.

MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition of unseen objects.

Jia-Xiang Fang, Shi-Qiang Ma, Jing Wang et al. · 0 citations
Preprint Oct 2026

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it...

Cigdem Kokenoz, A. Salarpour, Alkim Domeke et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.