Native Sparse Attention for Optical Remote Sensing Spectral Images
Abstract
Optical remote sensing (RS) spectral images contain rich cross-band response patterns. Recent RS foundation models (RSFMs) improve compatibility with variable-band inputs through unified or channelwise tokenization, but such input-interface compatibility does not ensure generalization to unseen sensors. Moreover, existing backbones often apply dense self-attention to flattened spatial–spectral tokens, resulting in high floating point operations per second (FLOPs) and memory costs while mixing heterogeneous interactions indiscriminately. To address this issue, we propose spectral-native sparse attention (SpectralNSA), a spectral-native sparse attention (NSA) backbone for optical RS spectral images. SpectralNSA models cross-channel tokens at each spatial location using three complementary sparse patterns: blockwise spectral compression for global context, group-shared Top-k block selection for informative spectral regions, and local spectral windows for neighboring-band continuity. We further introduce a spectral-shared-spatial (SSS) structure that compresses channelwise features into shared-spatial prototypes, performs compact spatial modeling, and reinjects spatial context through gated residual connections, thereby reducing redundant per-channel–spatial interactions. Extensive experiments on scene classification, urban and agricultural segmentation, hyperspectral driving-scene segmentation, and road extraction demonstrate consistent accuracy gains and favorable scalability, especially for moderate- and high-dimensional spectral inputs. A within-sensor variable-band evaluation further shows that a model trained with the complete channel set can directly process shortened wavelength-ordered sequences after unavailable bands are removed, without retraining. At a fixed spatial resolution, the FLOPs advantage of SpectralNSA over ChannelViT increases with the number of spectral channels, reaching $140.94\times $ for the Tiny model with 150 input channels. Controlled pretraining and transfer experiments further demonstrate its effectiveness for transferable spectral–spatial representation learning across classification and dense prediction tasks.