Vocabulary-Free Segmentation via Multi-mask Category Generation with MLLMs
Abstract
Mask-based recognition with multimodal large language models (MLLMs) offers a promising route to vocabulary-free segmentation. However, existing methods typically predict each candidate mask independently, limiting shared-context reasoning, while naive joint multi-mask generation suffers from cross-mask interference. We present VFSeg, a vocabulary-free segmentation framework for joint multi-mask recognition that predicts attribute-category outputs for all masks in a single pass. Specifically, Position-Anchored Embedding (PAE) is designed to establish stable positional anchors by binding observations of the same mask across crops to shared slots, thereby decoupling position from content. Building on this design, Attribute-Guided Disambiguation (AGD) further improves output discrimination by generating concise, visually grounded attributes to guide mask category prediction, thereby reducing confusion among similar targets. To support VFSeg training, we build VGAMask, a large-scale supervision corpus of mask-category-attribute tuples. Experiments show that VFSeg substantially outperforms prior vocabulary-free baselines and demonstrates stronger open-ended semantic capability.