GIRVFM: Taming Vision Foundation Models for Improved Generalization in Infrared Small Target Detection

Abstract

Infrared small target detection (IRSTD) aims to identify small and dim targets in infrared images and has become increasingly important in practical applications such as maritime surveillance, search and rescue, and remote sensing. Most of the IRSTD methods are typically designed for in-distribution settings and suffer significant performance degradation under out-of-distribution scenarios, limiting their applicability in real-world conditions. Existing efforts to improve the generalization of IRSTD models mainly rely on handcrafted architectural designs or feature-level constraints. On the contrary, vision foundation models (VFMs) have demonstrated remarkable generalization capabilities on natural images, yet their potential for IRSTD remains largely unexplored. To preserve the inherent generalization capability of VFMs while adapting them to infrared small target images, we propose GIRVFM. In particular, we freeze the entire VFM backbone with U-shaped connectivity to extract general representations and introduce a saliency and structure tokens enhancement mechanism (S2TEM) to adapt the backbone. S2TEM incorporates learnable saliency and structure tokens to capture region-level attention and structural information, enabling the VFM to better represent small infrared targets. In addition, a frequency-guided attention (FGA) is introduced in the auxiliary branch to extract frequency components corresponding to gradient differences between targets and background, thereby enhancing discriminative feature extraction and improving recognition of both targets and background through frequency-domain modulation of spatial features. Extensive cross-domain experiments on four benchmark datasets demonstrate that GIRVFM achieves state-of-the-art performance, significantly enhancing generalization in IRSTD tasks. Our code will be made public at https://github.com/MaYazhou224/GIRVFM