ConsistVIS: A Consistency-Aware Weakly Supervised Video Instance Segmentation Method

Abstract

Video Instance Segmentation (VIS) is a growing computer vision task that involves segmenting, categorizing, and tracking object instances across video frames. Traditional methods rely on fully-supervised training and computationally intensive transformer models, which are costly and limited by labeling requirements. To address these shortcomings, this paper proposes ConsistVIS, a novel weakly-supervised VIS model that emphasizes embedding vector consistency. ConsistVIS employs a bottom-up, discriminative strategy, focusing on deformation-based and discrimination-based modeling. Deformation-based modeling uses geometric knowledge and key region descriptions to track instance deformations over time. Discrimination-based modeling introduces a self-supervised criterion based on support vectors to evaluate embedding vector discriminability within clusters, guiding network parameters to generate meaningful instance-level embeddings. By decomposing the consistency problem into dynamically optimized sub-problems, ConsistVIS improves segmentation performance, interpretability, and controllability across datasets. Experiments on the YouTube-VIS 2019 dataset demonstrate its effectiveness, achieving AP scores of 48.0, 49.8, and 56.7 with R50, R101, and SwinL backbones, respectively, matching or exceeding fully-supervised counterparts with the R50 and R101 backbones. With the larger, transformer-based SwinL backbone, ConsistVIS narrows this gap relative to prior weakly-supervised methods, though fully-supervised approaches retain a modest edge, indicating that further work is needed for weak supervision to fully scale to high-capacity architectures.