Mitigating Task and Data Domain Discrepancy for Open- Vocabulary Video Instance Segmentation

Abstract

Current Open-Vocabulary Video Instance Segmentation (OVVIS) approaches rely on vision–language models (VLMs) for generalization to unseen categories. However, they often fuse segmentation queries with VLM features in a simple and direct manner, overlooking the domain discrepancy between these two tasks. This mismatch limits both the expressiveness of the queries and the zero-shot potential of VLMs. Moreover, most existing models keep the fixed queries learned from closed-set training data even when the test data distribution changes drastically, failing to enhance the language awareness of the model. To address these challenges, we propose Task and Data Domain-Aware Open-Vocabulary Video Instance Segmentation (TD-OVIS), a task-aligned query adaptation framework. Building upon a dual-branch approach, we improve open-vocabulary generalization by initializing queries with language guidance and classifying with test-time tuned text embeddings to mitigate the task and data domain gap, thereby aligning segmentation representations with the VLM embedding space. Extensive experiments on multiple benchmarks show that TD-OVIS generalizes well to unseen categories and achieves strong performance across multiple open-vocabulary VIS datasets.