Skip to content
Book Open access

Vocabulary-Free Segmentation via Multi-mask Category Generation with MLLMs

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 41 references

Abstract

Mask-based recognition with multimodal large language models (MLLMs) offers a promising route to vocabulary-free segmentation. However, existing methods typically predict each candidate mask independently, limiting shared-context reasoning, while naive joint multi-mask generation suffers from cross-mask interference. We present VFSeg, a vocabulary-free segmentation framework for joint multi-mask recognition that predicts attribute-category outputs for all masks in a single pass. Specifically, Position-Anchored Embedding (PAE) is designed to establish stable positional anchors by binding observations of the same mask across crops to shared slots, thereby decoupling position from content. Building on this design, Attribute-Guided Disambiguation (AGD) further improves output discrimination by generating concise, visually grounded attributes to guide mask category prediction, thereby reducing confusion among similar targets. To support VFSeg training, we build VGAMask, a large-scale supervision corpus of mask-category-attribute tuples. Experiments show that VFSeg substantially outperforms prior vocabulary-free baselines and demonstrates stronger open-ended semantic capability.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.