ALSCLIP: a symmetric encoder-decoder architecture integrating pre-trained CLIP and PointTransformer block for airborne LiDAR Scanning point cloud semantic segmentation
ABSTRACT The pretrain-finetune paradigm has the high-quality representation ability and transferability of their pretrained models, which has achieved great success in natural language processing (NLP) and 2D image fields. However, directly applying the pretrain-finetune paradigm to the point cloud field is still difficult due to the inherent domain gap between point cloud and image domains. In order to fully leverage the powerful representation learned from the large quantity of pretraining data, this paper proposes a novel neural network named ALSCLIP, which utilizes the large pretrained vision-language model CLIP as a pretrained backbone. Additionally, a symmetric encoder-decoder architecture and the Pointtransformer block were designed to learn general and transferable representations. To bridge the domain gap, we introduce a principled feature transfer mechanism that aligns low-level 3D geometric features from ALS point clouds with the high-level semantic embedding space of CLIP. Extensive controlled experiments are conducted to examine whether the frozen CLIP transformer, symmetric encoder-decoder architecture, and Pointtransformer block can serve as an efficient point cloud processing module on the 2019 IEEE GRSS Data Fusion Contest 3D Point Cloud Segmentation Challenge dataset and one engineering dataset in China. Cross-dataset generalization experiments further validate the transferability of the learned representations. The results show that ALSCLIP achieves state-of-the-art performance and can be widely used for ALS point segmentation. The source code of the proposed ALSCLIP network is available on https://github.com/Chenyang1112/ALSCLIP.