Multistage Uncertainty-Aware Feature Fusion Framework for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Abstract

Semantic segmentation of high-resolution remote sensing imagery is essential for land-cover mapping, urban planning, and environmental monitoring. However, challenges such as spectral similarity, fine-grained boundaries, and class imbalance often produce ambiguous predictions in complex scenes. Conventional encoder–decoder architectures fuse multilevel features uniformly, ignoring prediction uncertainty and allowing unreliable features to propagate through the decoding hierarchy, causing boundary misalignment and semantic confusion. To address these limitations, this study proposes a novel uncertainty-aware decoding framework that integrates prediction uncertainty into a progressive coarse-to-fine refinement process. The framework introduces two complementary decoder-side modules. First, an Uncertainty-Gated Channel Module (UGCM) leverages uncertainty derived from intermediate decoder predictions to recalibrate encoder skip features at the channel level prior to fusion, suppressing ambiguous low-level responses while preserving informative structural details. Second, a Multistage Uncertainty-Aware Fusion Module (mUAFM) performs adaptive feature fusion across multiple decoding stages by balancing high-level semantic confidence and low-level detail uncertainty. By alternating UGCM and mUAFM across decoding stages and employing multistage auxiliary supervision, the proposed decoder enables progressive coarse-to-fine prediction refinement guided by uncertainty while maintaining semantic consistency and boundary precision. Extensive experiments on three high-resolution benchmarks demonstrate superior performance over state-of-the-art methods, with notable improvements for spectrally ambiguous and under-represented classes. The framework achieves 57.06% mean intersection over union (mIoU) and 71.34% mean F1-score (mF1) on EarthVQA, 72.34% mIoU and 82.64% mF1 on Vaihingen, and 74.89% mIoU and 84.50% mF1 on Potsdam, with only 12.7M parameters. These results confirm the effectiveness of uncertainty-aware feature modulation and fusion for robust remote sensing segmentation.