Spatial and Frequency Dual-Domain Spatiotemporal Fusion Model With Convolution-Transformer for Remote Sensing Images

Abstract

Remote sensing images captured by single-sensors present the disadvantage of low spatial or temporal resolution, and spatiotemporal fusion (STF) is a convenient way of producing the high temporal- and spatial-resolution images essential for precise natural disaster monitoring. Available STF models remain challenging in limited prior data and abrupt changes of spatially heterogeneous scenarios, therefore, a spatial and frequency dual-domain STF model with convolution-Transformer for remote sensing images (SFCTM) is proposed, which produces fine-resolution images at the prediction date by exploiting a fine-resolution image at random date and a coarse-resolution image at the prediction date. Nonsubsampled contourlet transform is introduced to capture the high-frequency and low-frequency information of the fine-resolution image and the compound low-resolution image to preserve more favorable details and spectra. A frequency domain spatiotemporal-spectral feature encoder with convolution-Transformer is proposed to learn the local-global time-variant spectra and details in the frequency domain to enhance the prediction precision for scenarios with heterogeneous features and abrupt changes. The high-frequency cross-fusion and low-frequency cross-fusion provide superior aggregation and complementarity of dual-source high-frequency/low-frequency features, to further mitigate the bias of different sensors and preserve details and time-varying spectra. Finally, the convolution-Transformer decoder in the spatial domain is devised to reconstruct STF features and generate time-dense and high spatial-resolution images. Subjective and objective assessments of extensive experiments performed on well known coleambally irrigation area and lower gwydir catchment datasets demonstrate that the SFCTM yields superior prediction on abrupt change and spatially heterogeneous scenarios.