UnHOT: Toward Simple and Unified Hyperspectral Object Tracking

Abstract

Hyperspectral object tracking leverages its rich spectral material identification capabilities to address challenges that RGB video alone struggles to overcome, such as camouflaged targets and cluttered backgrounds. However, the high-dimensional spectral data introduces new challenges, including band gaps and substantial data volumes. In contrast to previous inefficient ensemble trackers or band selection methods that discard partial spectral information, this study proposes a simple, efficient, and unified hyperspectral tracking framework named UnHOT. Building upon a base spectral band group, we introduce a spatial-spectral cross-attention module to aggregate sparse residual spatial-spectral features, reducing the input data volume while accommodating varying numbers of spectral bands. This enables us to utilize cross-domain multi-dataset joint training and to handle HSVs with different numbers of bands using a unified structure and a single set of parameters. Additionally, to mitigate the scarcity of hyperspectral training data, we employ the LoRA-prompted frozen pre-trained transformer-based foundation model to learn robust, universal cross band domain visual representations. Moreover, to enhance inference speed, we propose a token fusion pruning strategy that discards uninformative tokens and merges tokens from base and residual band groups, striking a good trade-off between speed and accuracy. Extensive experiments and analyses validate the proposed framework, providing significant insights for future research. All codes will be made available at https://github.com/laisimiao/UnHOT