CARE: conflict-aware regulation and evolution for continual test-time open-vocabulary semantic segmentation

Abstract

Open-vocabulary semantic segmentation (OVSS) models provide strong zero-shot recognition, but their performance often degrades under continual target-domain shifts. Continual test-time adaptation for OVSS remains challenging because vision-language predictions are spatially coarse, prompted structural masks may contain incomplete or incorrect category assignments, and repeated updates can accumulate pseudo-label errors. To address these issues, we propose Conflict-Aware Regulation and Evolution (CARE), a continual test-time adaptation framework that updates only a small subset of model parameters. CARE consists of three complementary components. Spatial Conflict Resolution (SCR) evaluates class-conditioned masks generated by a frozen SAM3 model using mask confidence and semantic support, performs class-specific logit rectification, and rejects fusion that increases prediction entropy. Confidence-Driven Safe Evolution (CSE) restricts direct optimization to pixels with reliable pseudo-labels, thereby reducing noisy gradient updates. Conflict-Driven Plasticity (CDP) measures the full-image disagreement between the current prediction and the pseudo-label and uses it to determine whether the updated parameters should be retained or partially restored toward the zero-shot model. Extensive experiments show that CARE consistently outperforms existing TTA and OVSS adaptation baselines across multiple datasets and corruptions, while maintaining stable performance over extended and cross-corruption continual streams without introducing additional trainable modules.