Stratified Evaluation of SAM 2 for Zero-Shot Building Segmentation in Aerial Imagery

Abstract

: Building footprint extraction from remote sensing imagery underpins urban planning, population estimation, and disaster damage assessment. Deep learning methods have achieved high accuracy for this task, but their dependence on large-scale pixel-level annotations creates a severe bottleneck: annotating a city-scale dataset demands hundreds of hours of manual labor, limiting rapid deployment to new regions. The Segment Anything Model 2 (SAM 2), a foundation model with zero-shot segmentation capability, offers a potential solution by eliminating the need for task-specific annotations entirely. Yet SAM 2 was trained exclusively on natural scene images and videos, raising a critical question: can it generalize to the fundamentally different visual characteristics of aerial remote sensing imagery? This paper presents the first systematic zero-shot evaluation of SAM 2 for aerial building segmentation. We conduct three groups of experiments: (1) benchmarking four SAM 2 model variants to identify the optimal accuracy-efficiency trade-off; (2) stratified evaluation across dense, sparse, large-scale, and small-scale building morphologies to reveal which architectural characteristics challenge SAM 2 most; and (3) comparison of single-point, multi-point, and bounding-box prompting strategies to derive practical guidelines. Results demonstrate that SAM 2 Base+ achieves an IoU of 0.7738 without any training data, while oracle bounding-box prompting reaches 0.8755. SAM 2 excels on dense and large-scale buildings but struggles with sparse scenes. These findings establish SAM 2 as a viable tool for rapid building mapping while highlighting where domain adaptation remains necessary.