UST-Net: Unified Spatial–Spectral–Temporal Modeling for Adaptive Hyperspectral Object Tracking
Abstract
Hyperspectral (HS) imagery provides material-sensitive spectral cues that complement spatial appearance and improve target discrimination when red–green–blue (RGB) observations are ambiguous. Existing HS trackers have explored spectral prompting and temporal modeling, yet spectrum-preserving prompt construction, recurrent target-state propagation, and hierarchical feature adaptation remain insufficiently coordinated. We propose UST-Net, a unified spatial–spectral-- temporal framework that connects these processes within a shared transformer tracker. Its cross-modal prompt (CMP) module organizes co-registered false-color (FAC) and grouped multiband representations derived from the same HS cube into fixed-format prompts, preserving dependencies across spectral groups while accommodating different spectral configurations. The temporal state integration (TSI) module couples cross-frame Mamba hidden states with cross-stage semantic context and exchanges them with the prompt-enhanced token stream through paired injection and extraction operations. Conditioned on the resulting temporally refined tokens, the adaptive prompt injection (API) module generates stage-conditioned prompt context and applies it as residual modulation to downstream feature extraction. Experiments on the HOTC and HOT2024 datasets demonstrate that UST-Net achieves state-of-the-art performance across multiple spectral settings, with particularly strong results under occlusion and out-of-view (OV) conditions.