Jul 2026· The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences· Vol XLIX-B2-2026, pp. 223-229· 1 citation· 20 references
TL;DR
A zero-shot, open-vocabulary semantic segmentation framework for ALS point clouds based on 2D-3D transfer, utilizing three types of VFMs and introduces an adaptive global view projection module that derives optimal virtual camera poses and field-of-view (FOV) from scene extents, effectively enabling the application of 2D VFMs even in the absence of original imagery.
Abstract
Abstract. While deep learning has advanced ALS point cloud semantic segmentation and achieved impressive results, most methods rely on predefined label sets and lack ability to recognize arbitrary categories. Recently, the visual foundation models (VFMs) has garnered significant attention, due to remarkable zero-shot generalization capabilities by leveraging open-set knowledge. However, adapting these models to large-scale ALS point clouds remains largely unexplored and highly challenging. In addition, the frequent absence of well-aligned synchronously acquired images further hinders the application of 2D VFMs in ALS point clouds. To bridge these gaps, we developed a zero-shot, open-vocabulary semantic segmentation framework for ALS point clouds based on 2D-3D transfer, utilizing three types of VFMs. We employed a combination of VFMs, including source models pre-trained on natural imagery and models fine-tuned on remote sensing data, to investigate the generalization capabilities of VFMs in inherent domain gap between natural and aerial imagery. Besides, we further introduce an adaptive global view projection module that derives optimal virtual camera poses and field-of-view (FOV) from scene extents, effectively enabling the application of 2D VFMs even in the absence of original imagery. Quantitative evaluations on the Vaihingen dataset indicate that methods trained solely on natural images achieve segmentation accuracy scores of 72% (roof) and 59% (tree) for common classes but struggle with rare categories such as powerline. GSNET improves performance across most categories, highlighting importance of domain adaptation. Evaluation on the SUM dataset reveals that our approach effectively identifies large-scale urban elements (exceeding 60% precision for buildings) without high-quality, well-aligned imagery.
A SAM-guided framework for point cloud oversegmentation that significantly improves boundary recall and maintains high oracle accuracy while maintaining high oracle accuracy, and generalizes well to unseen datasets without retraining, showing strong cross-dataset inference capability.
Dening Lu, Michael A. Chapman, Jonathan Li· The International Archives o...· 0 citations
Generalized few-shot 3D point-cloud segmentation (GFS-PCS) asks a model to segment a scene into many base classes seen at training time and a set of novel classes. The state of the art reaches novel classes by reconciling a dense but noisy 3D vision-language prior with the few-shot support, but it pays for this with base 3D labels, per-episode training, and the support annotations themselves. We ask how far the same reconciliation can go with none of these: no training, no 3D labels, and not even the few-shot support. We pair a frozen 3D vision-language model (RegionPLC) as a dense prior with a frozen promptable concept segmenter (SAM3), prompted by the bare novel class names and lifted from posed RGB views, and reconcile the two by cross-view consistency: a point becomes novel only when enough of the views that see it agree. On the ScanNet200 GFS-PCS benchmark this fully training-free, open-vocabulary pipeline improves novel mIoU by +2.6 over the training-free dense prior while holding base accuracy within 0.5, and recovers a third (33%) of the novel-class gap to the trained state of the art that uses far more supervision. We further show that injecting the few-shot support into the pipeline, as a fusion gate and as a prototypical dense classifier, adds nothing over consistency alone and in fact degrades it through the classifier, which is why the method needs no support at all. On the harder ScanNet++ benchmark, where the dense prior is far weaker on novel classes, the same pipeline nearly doubles novel mIoU (+15.7, from 16.2 to 31.9) at a 1.7 base cost, lifting the harmonic mean from 21.5 to 31.1
Silas kwabla Gah, E. Owusu· arXiv.org· 0 citations
Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.
COSTA leverages the domain gap through proven test-time adaptation, and groups each batch of target-domain points into a small set of semantic clusters based on the similarity distribution in the adapted feature space, and propagates high-confidence pseudo labels obtained from an open-vocabulary vision-language model to all points through cluster-level voting.
Yanghong Lin, Li Fang, Tianyu Li et al.· 0 citations
DinoSplat-OV is proposed, a training-free framework that adapts DINOv3 to remote sensing without fine-tuning or additional pretraining, effectively filling the gap of DINO-series models in training-free open-vocabulary segmentation and providing a viable new path for further advances in this direction.
Changhao Zhao, Haoxiang Li, Yuke Li et al.· 0 citations
