A training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models and employs SAM-based models for accurate object mask generation is presented.
Abstract
In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.
The MeViS-Audio track asks a system to segment the objects described by a spoken motion expression throughout a video and to return empty masks when the described target is absent. We present a simple staged solution. Qwen3-ASR first converts speech into text. Several video mask tracks are then produced with complement...
Yi-Xuan Ren, Jia-Ning Liu, Ying-Xin Wang et al.· 0 citations
Across the nine leading solutions, foundation segmentation models are combined with target-aware memory, multimodal reasoning, explicit target-existence verification, agentic interaction, and corrective tracking, illustrating a broader transition from single-model mask propagation toward modular pipelines that reason a...
Chang Liu, Heng-Hui Ding, Ling-Yi Hong et al.· 0 citations
This report presents a two-stage, training-free solution for the MeViS-Text track of the 8th LSVOS Challenge, which ranked third on the challenge test set, obtaining J&F, J, F, N-acc.
Ce Bian, X. He, Jin-Rong Zhang et al.· 0 citations
A method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts, demonstrating flexible multimodal control and improved spatial consistency.
Tiernon Riesenmy, Zhang You, Gautam Bhattacharya et al.· 0 citations
STAG is introduced, to the authors' knowledge the first post-hoc framework for token-level spectro-temporal grounding of captions generated by audio-based MLLMs, and provides behavioral support for the faithfulness and selectivity of the explanations.
Lucia Cascone, V. Fraenza, Michele Nappi et al.· 0 citations
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evalu...
Jing Peng, Zi-Chao Nie, Zhi-Sheng Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
