Class Semantic Prototype Guided Fusion Network for Hyperspectral and LiDAR Data Classification

Abstract

Benefiting from information complementarity, the fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data for land cover classification has attracted significant attention in the remote sensing community. However, due to modality imbalance and information redundancy, effectively extracting and integrating complementary knowledge from HSI and LiDAR data remains a major challenge. To address the aforementioned issues, a class semantic prototype guided fusion network (CSPGFNet) is proposed to realize efficient and accurate classification by task-relevant feature mining, fusion, and interaction. Specifically, feature extraction sub-networks with multi-scale and multi-type convolutional structures are designed for each modality to mitigate semantic imbalance caused by inherent dimensional discrepancies. Moreover, a feature fusion-interaction module based on cross-attention mechanism is designed to fuse and interact task-relevant spatial–spectral and elevation information from modalities with class semantic prototype (CSP) as the bridge. Furthermore, a composite loss that incorporates multi-factor constraints is optimized to ensure information complementarity, semantic consistency and task relevance of the whole network. Experimental evaluations on three benchmark datasets demonstrate the effectiveness of the proposed method.