Hierarchically Decoupled Prompt Tuning with Cross-Domain Prediction Alignment for Unsupervised Domain Adaptation

Abstract

Prompt tuning offers an efficient way to adapt frozen vision-language models to unsupervised domain adaptation (UDA). Existing prompt-based UDA methods usually rely on domain-aware prompts, but they often couple category semantics and domain correction throughout the prompt hierarchy. This coupling can introduce redundant class encoding in deep prompts and limit the specialization of prompts for domain shift. We propose HDPT, a hierarchically decoupled prompt tuning framework for CLIP-based UDA. HDPT uses shallow prompts to anchor category semantics and domain identity, while deeper prompts discard explicit class prompts and focus on progressive domain correction. To couple the two modalities, shallow textual prompts are aggregated and projected to initialize the visual prompts, and layer-wise text-to-vision mappings are used in deeper layers. We further align source-conditioned and target-conditioned predictions with a Pearson correlation loss, which preserves cross-domain class-relation consistency without enforcing identical probability calibration. Experiments on Office-Home, VisDA-2017, mini-DomainNet, and Office-31 show that HDPT achieves competitive performance against recent prompt-based UDA methods, with clear gains on style-diverse transfer tasks. Code is available at https://github.com/zekaizhou/HDPT.git.