A novel mechanism to automatically identify which of these point-labels are suitable, and which are actively harmful, when used for propagation is introduced, paving the way for scalable ecological analysis.
Abstract
The health of marine ecosystems is a critical indicator of global environmental change, yet the physical constraints of underwater observation and the intrinsic challenges of processing marine imagery severely limit the scalability of systematic monitoring. While recent visual foundation models such as the Segment Anything Model (SAM) series show great promise, they still struggle with the fine-grained recognition required in these complex scenarios and still require expert supervision. Our work addresses this gap by bridging state-of-the-art foundation models with existing sparse supervision. Because historical benthic surveys are typically annotated with only a few sparse expert points per image, we utilize these legacy point-labels as visual prompts for SAM2. Our primary contribution is a novel mechanism to automatically identify which of these points are suitable, and which are actively harmful, when used for propagation. By filtering out unreliable points, we extract high-quality pseudo-ground-truth masks capable of training more accurate, fine-grained semantic segmentation models. We demonstrate the effectiveness of our approach on public benthic data and introduce a new, challenging benchmark featuring real-world sparse expert annotations, paving the way for scalable ecological analysis.
Abstract. Semantic segmentation of remote sensing imagery (RSI) is essential for urban mapping, land-use monitoring, and many other domains. However, pixel-level annotation is expensive, making weakly supervised semantic segmentation (WSSS) that relies on image-level labels an attractive alternative. Pre-trained models provide strong priors from large-scale learned representations, making them beneficial for WSSS. However, when kept frozen, they often produce sparse and misaligned class activation maps (CAMs) due to domain gaps and static inference. We propose a lightweight and efficient framework that integrates CLIP and DINO foundation models to address three challenges: (i) semantic misalignment between generic text prompts and RSI-specific visuals; (ii) static CAM quality; and (iii) incomplete object coverage. Our design includes: (1) a Textual Prototype-Aware Enrichment (TPE) module that builds an RS-specific knowledge base using large language model (LLM)-generated descriptions to enrich text prompts; (2) a Unified Semantic Relation Mining (USR) module that fuses learnable adapter features with CLIP attention and DINO affinity for online CAM refinement; and (3) a Visual Prototype-Aware Enrichment (VPE) module, which maintains momentum visual prototypes to complete regions and sharpen boundaries. By freezing the CLIP and DINO backbones and optimizing only lightweight adapter and decoder modules, the proposed framework reduces the number of trainable parameters while achieving competitive performance. Experimental on iSAID and ISPRS Potsdam datasets demonstrate the effectiveness of the proposed framework, achieving 38.01% mIoU on iSAID dataset and 47.01% mIoU with 66.89% overall accuracy on Potsdam dataset.
Xin Li, Nicola Genzano, M. Gianinetto et al.· ISPRS Annals of the Photogra...· 0 citations
Abstract. Ground based time lapse cameras provide continuous, high frequency observations of glacier dynamics; however, automated analysis of these image streams remains challenging due to fog, snowfall, lens contamination, and variable illumination. This study investigates the potential of adapting the foundation segmentation model Segment Anything Model 2 (SAM2) for glacier segmentation from ground-based monitoring. To enable integration into automated pipelines, SAM2 is configured in image mode with a learned prompt generation strategy, while fine-tuning is restricted to the prompt encoder and mask decoder. In addition, the internal Intersection over Union (IoU) prediction head is utilized as a confidence estimator to assess segmentation reliability. Experimental results demonstrate that the adapted model achieves stable segmentation under moderate environmental variability, while degrading under severe visibility loss. This stability is consistent across model scales and input resolutions. The confidence estimation further provides a meaningful signal for identifying uncertain predictions, supporting reliability-aware processing in downstream workflows.
Bindusara Nagathihalli Lokesh, Laura Camila Duran Vergara, Hans-Gerd Maas et al.· The International Archives o...· 1 citation
: Building footprint extraction from remote sensing imagery underpins urban planning, population estimation, and disaster damage assessment. Deep learning methods have achieved high accuracy for this task, but their dependence on large-scale pixel-level annotations creates a severe bottleneck: annotating a city-scale dataset demands hundreds of hours of manual labor, limiting rapid deployment to new regions. The Segment Anything Model 2 (SAM 2), a foundation model with zero-shot segmentation capability, offers a potential solution by eliminating the need for task-specific annotations entirely. Yet SAM 2 was trained exclusively on natural scene images and videos, raising a critical question: can it generalize to the fundamentally different visual characteristics of aerial remote sensing imagery? This paper presents the first systematic zero-shot evaluation of SAM 2 for aerial building segmentation. We conduct three groups of experiments: (1) benchmarking four SAM 2 model variants to identify the optimal accuracy-efficiency trade-off; (2) stratified evaluation across dense, sparse, large-scale, and small-scale building morphologies to reveal which architectural characteristics challenge SAM 2 most; and (3) comparison of single-point, multi-point, and bounding-box prompting strategies to derive practical guidelines. Results demonstrate that SAM 2 Base+ achieves an IoU of 0.7738 without any training data, while oracle bounding-box prompting reaches 0.8755. SAM 2 excels on dense and large-scale buildings but struggles with sparse scenes. These findings establish SAM 2 as a viable tool for rapid building mapping while highlighting where domain adaptation remains necessary.
Bingning Xiong, Mingyu Ou· Journal of image processing...· 0 citations
Monitoring aquatic biodiversity is vital for maintaining global ecological balance. While advancements in computer vision have revolutionized underwater perception, existing datasets are predominantly limited to coarse-grained categories or lack spatial localization annotations, severely constraining the applicability of models for fine-grained biological identification in real-world scenarios. To address this gap, we introduce OpenAqua, the first large-scale fine-grained dataset dedicated to open underwater visual tasks. OpenAqua is structured around a five-level biological taxonomic hierarchy, comprising 77,970 high-quality images covering 16,540 aquatic species, and providing 132,885 fine-grained bounding boxes and corresponding instance segmentation masks. Based on this dataset, we establish a comprehensive benchmark suite that encompasses not only standard object detection and instance segmentation tasks but also pioneers an underwater open-vocabulary object detection benchmark. Extensive experimental evaluations reveal a significant performance degradation in current models as they progress from coarse-grained to fine-grained recognition. These results highlight the substantial challenges associated with fine-grained semantic perception and domain adaptation in degraded underwater environments. We believe OpenAqua holds the potential to advance fine-grained underwater vision research, facilitate learning from long-tailed distributions, and enable more effective aquatic ecosystem monitoring. Our dataset is available at https://github.com/White-cat-ed/OpenAqua.
Linxuan Luo, Pan Mu, Cong Bai· Proceedings of the 32nd ACM...· 0 citations
Building extraction holds significant practical importance for urban planning and various human productive activities. However, the complex imaging mechanisms of remote sensing images (RSIs) and the inherent diversity of building features present substantial challenges to accurate extraction. Current mainstream research primarily relies on supervised learning or fine-tuning of foundation models. These approaches depend heavily on large volumes of meticulously annotated data, and thus suffer from high annotation costs and limited generalization capabilities. To overcome these limitations, an unsupervised, training-free framework exploiting pretrained visual foundation models is proposed for building extraction from high-resolution RSIs. This framework requires no human-annotated data, training, or fine-tuning, enabling direct zero-shot inference on high-resolution RSIs. Specifically, it first generates initial pseudolabels by adaptively fusing semantic features from DINO and dense prediction features from CLIP via a spatial correlation-guided weighting mechanism. Then, a dual-path optimization mechanism based on the segment anything model (SAM) is introduced to refine these pseudolabels: local refinement corrects building boundaries, while global filtering enhances regional integrity. This dual-path design, tailored to the geometric characteristics of buildings, is the first attempt to fully exploit SAM’s complementary capabilities in a unified training-free pipeline. Experiments on three public datasets (WHU, WHU-Mix, and Inria) demonstrate that the proposed method achieves F1 scores of 71.86%, 61.48%, and 57.90%, respectively. This performance significantly surpasses current state-of-the-art unsupervised methods, exhibiting excellent accuracy, robustness, and cross-dataset adaptability, and providing valuable insights for practical remote sensing applications.
Junming Chen, Bing Liu, Weiqi Lian et al.· IEEE Journal of Selected Top...· 0 citations
Abstract. Accurate building segmentation from high-resolution aerial imagery is essential for various urban applications such as digital twins, geographic information system (GIS), and flood risk modelling. However, conventional supervised deep learning approaches require large amounts of pixel-level annotations, which are costly and time-consuming to obtain for large remote sensing datasets. To address this limitation, self-supervised learning (SSL) has recently emerged as an effective paradigm for learning visual representations from unlabeled data. In particular, masked autoencoders (MAE) have demonstrated strong performance by reconstructing masked image patches during pretraining. Nevertheless, conventional MAE frameworks rely on random masking strategies that ignore the spatial structure and semantic importance of regions in high-resolution remote sensing imagery. In this study, we propose GeoRGMAE, a geospatially guided masked autoencoder pretraining strategy for building segmentation. Unlike standard MAE, which rely on random masking, our approach leverages building footprint annotations available during pretraining to guide the masking process while preserving the original reconstruction objective. We introduce three masking strategies -core, balanced, and density-aware masking- that prioritize semantically relevant building regions under the varying urban densities. The core strategy focuses on building interiors, the balanced strategy distributes masking between buildings and background, and the density-aware adapts masking based on scene-level building density. Experiments on the Roof3D and WHU Building datasets demonstrate consistent, though modest, improvements over standard MAE pretraining, with the most effective masking strategy depending on dataset characteristics. These findings suggest that incorporating geospatial priors into masked image modelling (MIM) can improve representation learning for downstream building segmentation tasks.
Tuğba Eraslanoğlu, G. Mutreja, Martin Kada et al.· The International Archives o...· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
